English | 简体中文 | 中国镜像 |
💖 PulsarRPA is All You Need! 💖
PulsarRPA is a high-performance, distributed, open-source Robotic Process Automation (RPA) framework. It is designed to handle large-scale RPA tasks with ease, providing a comprehensive solution for browser automation, web content understanding, and data extraction. It addresses the inherent challenges of browser automation and extracting accurate, comprehensive web data amidst rapidly evolving and increasingly intricate websites.
val session = PulsarContexts.createSession()
val document = session.harvest("https://www.eeo.com.cn/2024/0330/648712.shtml")
println(document.contentTitle)
println(document.textContent)
Example code: kotlin.
In this scenario, we employed the popular boilerpipe technology to automatically extract content from news webpages. However, PulsarRPAPro offers more advanced data extraction techniques that can automatically handle a wider variety of webpages, including product detail pages.
fun main() = PulsarContexts.createSession().scrapeOutPages(
"https://www.amazon.com/", "-outLink a[href~=/dp/]", listOf("#title", "#acrCustomerReviewText"))
The code above scrapes fields specified by CSS selectors #title and #acrCustomerReviewText from a set of product pages.
Example code: kotlin.
fun main() {
val context = PulsarContexts.create()
val parseHandler = { _: WebPage, document: FeaturedDocument ->
// use the document
// ...
// and then extract further hyperlinks
context.submitAll(document.selectHyperlinks("a[href~=/dp/]"))
}
val urls = LinkExtractors.fromResource("seeds10.txt")
.map { ParsableHyperlink("$it -refresh", parseHandler) }
context.submitAll(urls).await()
}
val options = session.options(args)
val event = options.event.browseEventHandlers
event.onBrowserLaunched.addLast { page, driver ->
// warp up the browser to avoid being blocked by the website,
// or choose the global settings, such as your location.
warnUpBrowser(page, driver)
}
event.onWillFetch.addLast { page, driver ->
// have to visit a referrer page before we can visit the desired page
waitForReferrer(page, driver)
// websites may prevent us from opening too many pages at a time, so we should open links one by one.
waitForPreviousPage(page, driver)
}
event.onWillCheckDocumentState.addLast { page, driver ->
// wait for a special fields to appear on the page
driver.waitForSelector("body h1[itemprop=name]")
// close the mask layer, it might be promotions, ads, or something else.
driver.click(".mask-layer-close-button")
}
// visit the URL and trigger events
session.load(url, options)
Example code: kotlin.
select
dom_first_text(dom, '#productTitle') as title,
dom_first_text(dom, '#bylineInfo') as brand,
dom_first_text(dom, '#price tr td:matches(^Price) ~ td, #corePrice_desktop tr td:matches(^Price) ~ td') as price,
dom_first_text(dom, '#acrCustomerReviewText') as ratings,
str_first_float(dom_first_text(dom, '#reviewsMedley .AverageCustomerReviews span:contains(out of)'), 0.0) as score
from load_and_select('https://www.amazon.com/dp/B0C1H26C46 -i 1s -njr 3', 'body');
Example code:
Extracting Web data at scale is extremely hard. Websites change frequently and are becoming more complex, meaning web data collected is often inaccurate or incomplete. PulsarRPA has developed a range of cutting-edge technologies to solve this problem.
To conquer these challenges, PulsarRPA incorporates a suite of innovative technologies that ensure efficient, accurate, and scalable web data extraction:
In summary, PulsarRPA, with its web content understanding, intelligent scraping, advanced DOM parsing, distributed processing, and open-source features, becomes the preferred open-source solution for large-scale web data extraction. Its unique technology combination allows users to effectively address the complexities and challenges associated with extracting valuable web data on a large scale, ultimately facilitating wiser decision-making and competitive advantage.
We have released complete solutions for site-wide Web scraping for some of the largest e-commerce websites. These solutions meet the highest standards of performance, quality, and cost. They will be free and open source forever, such as:
🕷 PulsarRPA supports high-quality, large-scale Web data collection and processing. We have developed a range of infrastructure and cutting-edge technologies to ensure the highest standards of performance, quality, and TCO (total cost of ownership), even in very large-scale data collection scenarios.
🏪 PulsarRPA supports the Network-As-A-Database paradigm. PulsarRPA treats the external network as a database. If the required data is not in the local storage, or the existing version does not meet the analysis needs, the system will collect the latest version of the data from the Internet. We also developed X-SQL to query the Web directly and convert webpages into tables and charts.
🌈 PulsarRPA supports browser rendering as the primary method to collect Web data. By using browser rendering as the primary method to collect Web data, we achieve an optimal balance between data point scale, data quality, labor cost, and hardware cost, and achieve the lowest TCO (total cost of ownership). With optimizations such as blocking unnecessary resource files, the performance of browser rendering can even be comparable to the traditional single resource collection method.
💫 PulsarRPA supports RPA based Web scraping. PulsarRPA includes an RPA subsystem for Web interaction: scrolling, typing, screen capture, dragging and dropping, clicking, etc. This subsystem is similar to the well-known selenium, playwright, puppeteer, but all behaviors are optimized, such as more realistic simulation, better execution performance, better parallelism, better fault tolerance, and so on.
🔪 PulsarRPA supports single resource collection. PulsarRPA’s default data collection method is to harvest the complete
Web data through browser rendering, but if the data you need can be retrieved through a single link, for example, it can be returned through an ajax interface, you can also call PulsarRPA’s resource collection method for super High-speed collection.
💯 PulsarRPA plans to support cutting-edge information extraction technology. We plan to release an advanced AI to automatically extract every field from all valuable webpages (e.g., product detail pages) with remarkable accuracy, and we currently offer a preview version.
To unlock the full potential of PulsarRPA and tackle even the most complex data scraping tasks, a solid understanding of its core concepts is essential. By grasping these fundamental principles, you’ll be equipped to wield PulsarRPA as a powerful tool for extracting valuable information from the web. Let’s delve into the key concepts that form the foundation of your data scraping journey with PulsarRPA:
load arguments
. These control parameters fine-tune how PulsarRPA loads and scrapes webpages, allowing you to tailor the process to specific requirements.Check PulsarRPA concepts for details.
We have released a standalone executable jar based on PulsarRPA, which includes:
self-supervised
machine learning for information extraction, AI identifies all fields on the detail page with over 90% field accuracy of 99.9% or moreself-supervised
machine learning and outputs all extract rules, which can help traditional Web scraping methodsDownload PulsarRPAPro and explore its capabilities with a single command line:
java -jar exotic-standalone.jar
The simplest way to leverage the power of PulsarRPA is to add it to your project as a library.
Maven:
<dependency>
<groupId>ai.platon.pulsar</groupId>
<artifactId>pulsar-all</artifactId>
<version>1.12.9</version>
</dependency>
Gradle:
implementation("ai.platon.pulsar:pulsar-all:1.12.9")
You can clone the template project from github.com: kotlin, java-11, java-17.
You can also start your own large-scale web crawling projects based on our commercial-grade open source projects: PulsarRPAPro, Exotic-amazon.
Check the quick start for more details.
When PulsarRPA runs as a REST service, X-SQL can be used to scrape webpages or to query web data directly at any time, from anywhere, without opening an IDE.
git clone https://github.com/platonai/pulsar.git
cd pulsar && bin/build-run.sh
For Chinese developers, we strongly suggest you to follow this instruction to accelerate the building process.
Start the pulsar server if it is not started:
bin/pulsar
Scrape a webpage in another terminal window:
bin/scrape.sh
The bash script is straightforward. It merely uses curl to send a POST request with an X-SQL.
curl -X POST --location "http://localhost:8182/api/x/e" -H "Content-Type: text/plain" -d "
select
dom_base_uri(dom) as url,
dom_first_text(dom, '#productTitle') as title,
dom_first_slim_html(dom, 'img:expr(width > 400)') as img
from load_and_select('https://www.amazon.com/dp/B0C1H26C46', 'body');
"
Example code: bash, batch, java, kotlin, php.
Click X-SQL to see a detailed introduction and function descriptions about X-SQL.
We have a step-by-step course by example:
PulsarRPA has carefully designed the logging and metrics subsystem to record every event that occurs in the system. PulsarRPA logs the status for every load execution, providing a clear and comprehensive overview of system performance. This detailed logging allows for quick assessment of the system’s health and efficiency. It answers key questions such as: Is the system operating smoothly? How many pages have been successfully retrieved? How many attempts were made to reload pages? And how many proxy IP addresses have been utilized? This information is invaluable for monitoring and troubleshooting purposes, ensuring that any issues can be promptly identified and addressed.
By focusing on a concise set of indicators, you can unlock a deeper understanding of the system’s overall condition: 💯 💔 🗙 ?💿 🔃 🤺.
Typical page loading logs are shown below. Check the log-format to learn how to read the logs and gain insight into the state of the entire system at a glance.
2022-09-24 11:46:26.045 INFO [-worker-14] a.p.p.c.c.L.Task - 3313. 💯 ?U for N got 200 580.92 KiB in 1m14.277s, fc:1 | 75/284/96/277/6554 | 106.32.12.75 | 3xBpaR2 | https://www.walmart.com/ip/Restored-iPhone-7-32GB-Black-T-Mobile-Refurbished/329207863 -expires PT24H -ignoreFailure -itemExpires PT1M -outLinkSelector a[href~=/ip/] -parse -requireSize 300000
2022-09-24 11:46:09.190 INFO [-worker-32] a.p.p.c.c.L.Task - 3738. 💯 💿 U got 200 452.91 KiB in 55.286s, last fetched 9h32m50s ago, fc:1 | 49/171/82/238/6172 | 121.205.220.179 | https://www.walmart.com/ip/Boost-Mobile-Apple-iPhone-SE-2-Cell-Phone-Black-64GB-Prepaid-Smartphone/490934488 -expires PT24H -ignoreFailure -itemExpires PT1M -outLinkSelector a[href~=/ip/] -parse -requireSize 300000
2022-09-24 11:46:28.567 INFO [-worker-17] a.p.p.c.c.L.Task - 2269. 💯 🔃 U for SC got 200 565.07 KiB <- 543.41 KiB in 1m22.767s, last fetched 16m58s ago, fc:6 | 58/230/98/295/6272 | 27.158.125.76 | 9uwu602 | https://www.walmart.com/ip/Straight-Talk-Apple-iPhone-11-64GB-Purple-Prepaid-Smartphone/356345388?variantFieldId=actual_color -expires PT24H -ignoreFailure -itemExpires PT1M -outLinkSelector a[href~=/ip/] -parse -requireSize 300000
2022-09-24 11:47:18.390 INFO [r-worker-8] a.p.p.c.c.L.Task - 3732. 💔 ?U for N got 1601 0 <- 0 in 32.201s, fc:1/1 Retry(1601) rsp: CRAWL, rrs: EMPTY_0B | 2zYxg52 | https://www.walmart.com/ip/Apple-iPhone-7-256GB-Jet-Black-AT-T-Locked-Smartphone-Grade-B-Used/182353175?variantFieldId=actual_color -expires PT24H -ignoreFailure -itemExpires PT1M -outLinkSelector a[href~=/ip/] -parse -requireSize 300000
2022-09-24 11:47:13.860 INFO [-worker-60] a.p.p.c.c.L.Task - 2828. 🗙 🗙 U for SC got 200 0 <- 348.31 KiB <- 684.75 KiB in 0s, last fetched 18m55s ago, fc:2 | 34/130/52/181/5747 | 60.184.124.232 | 11zTa0r2 | https://www.walmart.com/ip/Walmart-Family-Mobile-Apple-iPhone-11-64GB-Black-Prepaid-Smartphone/209201965?athbdg=L1200 -expires PT24H -ignoreFailure -itemExpires PT1M -outLinkSelector a[href~=/ip/] -parse -requireSize 300000
java
on the PATHPulsarRPA is tested on Ubuntu 18.04, Ubuntu 20.04, Windows 7, Windows 11, WSL, and any other operating system that meets the requirements should work as well.
Check the advanced topics to find out the answers for the following questions:
In general, the features mentioned in the Feature section are well-supported by PulsarRPA, but other solutions do not.
Check the solution comparison to see the detailed comparison to the other solutions:
Check the technical details to see answers for the following questions: