How can you download a product data feed?

[Honeybadger]

Is there a way to download the the product data feed of a public website that you don't have back end access to (without doing anything shady)?

 

[T J Tutor]

Many sites offer access to their data feeds, rss feeds, and news feeds. You just have to look for the option, usually noted in their footer.

 

[Graybeard]

in a LINUX terminal

$ wget -N [URL]
$ curl
$man curl
#read the manual
#create a crontab
#parse the data

 

[Honeybadger]

in a LINUX terminal

$ wget -N [URL]
$ curl
$man curl
#read the manual
#create a crontab
#parse the data

Appreciated
Is there a YT video with instructions?

 

[adnanahsan]

If your products provider site provide any feed service you may grab the data easily, usually mostly companies provide data in json/xml or a structured array.

 

[Honeybadger]

Appreciate the replies
The question is can it be done front end
As in can you grab a product data feed direct from a website?

 

[Honeybadger]

I never heard of this Czechia, typo?

 

[Graybeard]

Depends on the feed -- if it's in HTML you could just include it
If it's JSON or XML/rss (or csv) you have to parse it into HTML.

 

[heitravel]

To download a product data feed, follow these general steps:

1. Check if the retailer or supplier offers a product data feed download option. Some retailers or suppliers may offer product data feeds that can be downloaded directly from their websites or through a third-party service.
2. If the retailer or supplier offers a download option, locate the link to the product data feed. This may be found in a dedicated section of the website, such as the "Downloads" or "Resources" section.
3. Click on the link to download the product data feed. The file may be downloaded in a compressed format such as ZIP or GZIP.
4. Extract the downloaded file to access the product data feed. Some product data feeds may be in CSV, XML or other file formats, so make sure you have the appropriate software to open and read the file.
5. If there is no download option available, contact the retailer or supplier and inquire about the availability of a product data feed. They may be able to provide you with access to the data through another means, such as an API or FTP site.

 

[Graybeard]

you can parse a portion of a product feed into some segment --a segment like men's suede shoes or music by an artist/band (possibilities are endless). You need to create a valid .json file.
data=[{"artist":"name","title":"song"},{"key":"value","K":"V"},{"":"","":""}];
You can print this to a JavaScript to parse it.

<!-- Include the JavaScript file
for Bootstrap table -->
<script src=
"https://unpkg.com/bootstrap-table@1.16.0/dist/bootstrap-table.min.js">
</script>
<script type="text/javascript">
    $(document).ready(function () {

    // Use the given data to create
    // the table and display it
    $('table').bootstrapTable({
        data: mydata
    });
    });

    // Specify the JSON data to be displayed
    var mydata =[ see example above];
</script>

then the HTML to make the table.

Super fast without a database query and response

 

[T J Tutor]

To download a product data feed, follow these general steps:

1. Check if the retailer or supplier offers a product data feed download option. Some retailers or suppliers may offer product data feeds that can be downloaded directly from their websites or through a third-party service.
2. If the retailer or supplier offers a download option, locate the link to the product data feed. This may be found in a dedicated section of the website, such as the "Downloads" or "Resources" section.
3. Click on the link to download the product data feed. The file may be downloaded in a compressed format such as ZIP or GZIP.
4. Extract the downloaded file to access the product data feed. Some product data feeds may be in CSV, XML or other file formats, so make sure you have the appropriate software to open and read the file.
5. If there is no download option available, contact the retailer or supplier and inquire about the availability of a product data feed. They may be able to provide you with access to the data through another means, such as an API or FTP site.

THIS IS 100% AI AND IS A VIOLATION OF OUR TOS!
STOP POSTING AI CONTENT!

 

[Graybeard]

this is sooo aggravating sometimes
I am using something called PJAX
*note the loaded time printed on the left and the right columns --they are different because:
I am just replacing the <div id="content"></div> of my wrapper (the header, menu and the right
<div class col-4-md>)
Why super fast load --the wrapper is 'static' (more or less --now) the content div is dynamic --it will change with each page.

Visitor from $countryName is a IP database lookup done locally (max mind geo-ip-city.mdb)

The data is from https://coronavirus-19-api.herokuapp.com/countries updated every 2 hours. I parse the data I want to a new json file.

The file is based on the CIA PPP data +- my edits --PPP is comparable personal purchasing power (what people earn and what consumer items cost [supposedly])
So, how is you country doing (if it's listed?)
**the printed data will per parsed into a table with a JavaScript

Country

Deaths:1Million

The empty column on the right is for ads and affiliate offers for the traffic that will be on this site.

 

[T J Tutor]

The solution is knowing what to ask --as usual

EXACTLY! One has to know the question! The proper question.

 

[MarkTwels]

One common approach is to check if the website offers an API (Application Programming Interface) that allows access to their data. Many websites provide APIs for developers to retrieve data in a structured and organized manner.

If there's no API available, you can consider web scraping techniques. However, it's essential to be respectful of the website's terms of service and policies. Always make sure you're not overloading their servers with requests or violating any legal restrictions.

For more tips on data-related topics, you can take a look at this link It might provide you with some useful insights.

 

[Graybeard]

Nope it's CZ --that is the nation's Czech name not the anglicized Czech Republic. That is the common name now.
BTW Czechia was 599 two weeks ago, IIRC, they have a major problem there apparently with COVID-19 fatalities.

 

[Graybeard]

so I couldn't get jQuery to play nice with PJAX so I changed the format to HTML and changed the cron *.sh to make that file too

jQuery issue was the Loading please wait the table was fully loaded so why the wait? (error) Ather a few hours troubleshooting --no answer So tomorrow is table css time ...
I have about 4,000 pages to script so this is the prototype at this point ...

The table on the webpage

the HTML Source code

Long irritating day because of the flukes in using PJAX

• PJAX makes your web page load and feel like an app without all that development time and/or expense.
• Each time it reloads the content <div id='content'></div> the <title> is changed on the wrapper.
• You can also change any other meta tag also.
  <description> as an example.
  So, the SEO is great for a content feed or dynamic site.

However, first you need the data or feeds to work with --that is always an issue.

As an aside --the jQuery json parse works very well with a static web page --just requires some pasting of the json data. XML or CSV can be made into json with PHP or a Bash Script *.sh. So, any feed can be worked with.

Just need the feed

 

[Graybeard]

well good enough for hand grenades (as they say)

 

[Graybeard]

@heitravel If you knew WTF you were doing AI might have been useful.

The solution is knowing what to ask --as usual

the command is:

$ crontab -e
#enter as the user or sudo bash to edit root's crontab
#no line breaks in the cron and there need to be an empty line at the end of the crontab file.
#You control panel may have this function in a web dialog page "Cron"

 

[Honeybadger]

Downloading a product data feed typically depends on the source from which you want to obtain the data. Here are general steps you can take:

1. Access the Data Feed:
   • Identify the platform or website that provides the product data feed. It could be an e-commerce website, a supplier's portal, or any other source that offers product information in a structured format.
2. Check for Available Options:
   • Look for options provided by the platform to download the product data feed. Some platforms may offer a direct download link, while others might require you to access the data through an API or a specific interface.

got it
needs a programmer
it can be done on the front end
sitemap.xml probably the first step

 

[Graybeard]

There are plugins for WordPress that will parse a data feed.
However, this is the problem (either way,) the duplicate content from that feed being used verbatim on many, usually affiliate sites.

You could only use the item's image and rework the text (content/ description) as static text or programmatically altered. Changing text programmatically can create other issues with SEO --as the text changes the rankings may change also.