Data scraping with node js and display in laravel

Hello, Everyone laravelcode today share somthing new. we are working some day ago with one laravel application and we are required some another site data to in our laravel application using data scriping. so, we are done our data scriping logic in nodejs and data display logic in laravel. here we are share one simple example demo for data scraping with nodejss and how to display in laravel application

Today, in IT world day by day required something do new. but sometime some functionality code write into php is very difficult(but not imposible) and we are write that code login in another technology and then after merge. this method is very good for better performance for site.

Nodejs and laravel currently most famous framework for web-development and today many project develope using nodejs and laravel in one platform. for example real time chat system is best example for it.

Many laravel developer did't knowing how to use nodejs in laravel application. so we are share here one tutorials may be it's more helpfull to you for how to start nodejs setup in laravel and some basic stuff

We are starting here to install node and how to use with laravel application step by step

Our scraping data dispay preview look like

Step : 1 Install node js

For this tutorial we are required node so first we are install node in our local system. sumple run following command in your terminal.

sudo apt-get update
sudo apt-get install nodejs
sudo apt-get install npm

Check node version node -v

Check npm version npm -v

Step : 2 Create one laravel application

After install node js then we are required creae one laravel dummy application run by following command.

composer create-project --prefer-dist laravel/laravel LaravelWithNodeJs

After successfully create laravel aplication then configure you database, user and password in .env file.

Step : 3 Install npm package dependency

Open your laravel project root directory's package.json file and add following dependency in this file like that.

"devDependencies": {
    "express"    : "latest",
    "request"    : "latest",
    "cheerio"    : "latest"

After add above dependency then after run following command to install it in our laravel project directory

sudo npm install

After run this command then check in your project directory here created node_modules folder automatic and your all npm and node dependency installed there.

Step : 4 Check node js working

Now how to check our node js code working in our application or not. so first create file node/server.js and simply write followign code for testing.

var express = require('express');
var app     = express();

console.log('Your node server start....');

exports = module.exports = app;

Then after goto your project_derectory/node path and run folloeing command

sudo node server.js

If after run this command and you sow "Your node server start...." this output string then your node js code perfect work/run in your local system.

Step : 5 Write nodejs scraping code

Now we are write our target site scraping code in nodejs. here we are targeting this website for demo.

Before starting write nodejs scraping script first show following html structure which use in our targeting site

<td class="title">
	<a href="link" class="storylink">
	<span class="sitebit comhead"> (
	<a href="from?">
	<span class="sitestr"></span>

In this nodejs scraping code we are targetin span.comhead html dome and scraping following structured data

  • 1) rank
  • 2) title
  • 3) url
  • 4) points
  • 5) username
  • 6) comments

Now open your node/server.js file and put following nodejs scraping code.

var express = require('express');
var fs = require('fs');
var request = require('request');
var cheerio = require('cheerio');
var app     = express();

//Scraping start
app.get('/scrape', function(req, res){

	request('', function (error, response, html) {
	  	if (!error && response.statusCode == 200) {
	    	var $ = cheerio.load(html);
	    	var parsedResults = [];
	    	$('span.comhead').each(function(i, element){
	      		// Select the previous element
	      		var a = $(this).prev();
		      	// Get the rank by parsing the element two levels above the "a" element
		      	var rank = a.parent().parent().text();
		      	// Parse the link title
		      	var title = a.text();
		      	// Parse the href attribute from the "a" element
		      	var url = a.attr('href');
		      	// Get the subtext children from the next row in the HTML table.
		      	var subtext = a.parent().parent().next().children('.subtext').children();
		      	// Extract the relevant data from the children
		      	var points = $(subtext).eq(0).text();
		      	var username = $(subtext).eq(1).text();
		      	var comments = $(subtext).eq(2).text();
		      	// Our parsed meta data object
	      		var metadata = {
	        		rank: parseInt(rank),
	        		title: title,
	        		url: url,
	        		points: parseInt(points),
	        		username: username,
	        		comments: parseInt(comments)
	      		// Push meta-data into parsedResults array
	    	// Log our finished parse results in the terminal

	  	fs.writeFile('../public/output.json', JSON.stringify(parsedResults, null, 4), function(err){
	    	console.log('Sraping data successfully written! - Check your project public/output.json file');

	  	res.send('Scraping Done...');


console.log('Your node server start successfully....');

exports = module.exports = app;

Then after goto your project_derectory/node path and run folloeing command

sudo node server.js

After starting node server then open your browser and type following url


after run this url in browser you got this message in browser window"Scraping Done..." now your data scraping done. then check your terminal here you also getting one message like thatSraping data successfully written! - Check your project public/output.json file

Now we are done data scraping proccess and our data scraping file automatic write in json formate data in public/output.json file. now how to display this data in our laravel application front site. simple follow this step.

Step : 6 Create route

First create one route for a view/blade file like that

Route::get('data-scraping', 'DataScrapingController@index');

Step : 7 Create controller

now create DataScrapingController.php file look like

namespace App\Http\Controllers;

use App\Http\Requests;
use Illuminate\Http\Request;
use DB;
use Session;

class DataScrapingController extends Controller
    public function index()
        $data = json_decode(file_get_contents('output.json'));

        return view('data-scraping', compact('data'));

Step : 8 Create blade/view file

Into the last create resources/views/data-scraping.blade.php file and write html code like that for display data.


<div class="container">
	@foreach($data as $key => $value)
	<div class="col-sm-12">
      	<a href="{!! $value->url !!}">
      		<h3 class="title">{!! $value->title !!}</h3>
      	<p class="text-muted">
	      	<strong>Points :</strong> {!! $value->points !!}
	      	<strong>Comments :</strong> {!! $value->comments !!}
      	<p class="text-muted">Posted by <a href="#">{!! $value->username !!}</a></p>

Now we are ready to run our example so run bellow command ro quick run:

php artisan serve

Now you can open bellow URL on your browser:


We are hope it can help you...

 Prev post

Next post