Need Advice on Datafeeds

[Silencio]

Hello,

I am new to this forum and I am here because I need advice, help and guidance with adding datafeeds to my sites so I am hoping that I will find some help here on this wonderful database related forum......

Background: I have been building webpages for about 8-9 years now and I use Dreamweaver in addition to using WordPress in the Semiologic Pro version. I have pages in asp and php but have really come to love php and WordPress.

I am not a programmer but have dabbled with it a little, very little and am not that good at it but I can usually figure things out or find where who can help me.......

I have recently purchased RSS2Blog and have just uploaded my first datafeed from CJ to a WordPress Blog.

I also have recently purchased Teli's DatafeedtoRSS program. Teli if you are around I just want you to know that I have sent you a few emails through your web page over the last few months with no response....so I guess you didn't get them.....but maybe you can help me here on this forum if I need it.

I have a few questions that I still have not figured out yet.....and here they are related to the programs that I have currently.

1. When using a datafeed how often should you update it if you are posting prices? if you are not showing prices?

2. If I am using a program like RSS2Blog to upload your datafeds how do you "Update" the feeds without duplicating them? Might seem like a simple question to the more experienced but I just need to get a handle on this.....thanks please go easy on me.....

3. How would you, using RSS2Blog, set up pages with multiple products using small icons and then have individual pages on the products with larger photos available?

4. When using DatafeedtoRSS and using the "Import" feature on WordPress is there a way to "schedule" the posts to be random? I am thinking not but wondered in case there was....

Thanks much for any help! It is much appreciated

I guess I am also trying to learn more about how to customize and tweak my pages with datafeeds on them......

Charlie

 

[Linda Buquet]

Hi Charlie,

Welcome to 5 Star. I'll poke Teli to be sure she sees this thread as she can give you the most help probably.

Regarding how often to update. Partly depends on the merchant and how often their stock and/or pricing changes. Even if you don't show prices you want to keep up to date so you don't send a visitor to a product page that's out of stock or re-directs to the homepage.

Ideally you should try to set up a chron job to auto update for you so you don't have to do it manually. I would think you would want to update at least weekly if you do it manually but if it's automatic maybe daily depending on merchant???

I'm not sure about this advice because I don't get as many datafeed questions lately so have not kept up with things as much as I used to on this front.

 

[Teli]

Teli if you are around I just want you to know that I have sent you a few emails through your web page over the last few months with no response....so I guess you didn't get them.....but maybe you can help me here on this forum if I need it.

Actually, I haven't received them. Now this has me worried that I'm not receiving others' e-mails if they use the contact form on the site, because I set it up to always deliver the e-mails to me...I tout it as the most reliable way to reach me (sending a message via the contact form).

If you don't mind, would you help me figure out why your messages aren't coming through? First, let me know which form you're using to try and contact me (direct link is helpful), the e-mail address you're using (feel free to PM me with it so you don't get on any spambot radar), and whether or not you received any errors when submitting the form.

In the meantime, of course you can get help from me in the forums. Or you can contact me through any of my other contact forms as well.

1. When using a datafeed how often should you update it if you are posting prices? if you are not showing prices?

Depends on how often the merchant updates.

2. If I am using a program like RSS2Blog to upload your datafeds how do you "Update" the feeds without duplicating them?

Unfortunately, I've only used RSS2Blog a few times for other clients so cannot properly answer your question.

I can say that WP's built-in RSS import utility will not republish an entry that has the same title. So, if the datafeed has been updated and the title of the product remains the same, WP wouldn't import it again...however, it also wouldn't update the entry with information that's changed either.

Unfortunately, you may need to contact Michelle for an answer to this question. I know on the WPDFI script, there's a setting to "overwrite" the information of previous imports, so there may be something similar in RSS2Blog.

3. How would you, using RSS2Blog, set up pages with multiple products using small icons and then have individual pages on the products with larger photos available?

Well, again, I'm not well versed with RSS2Blog. Also, I don't know whether it would support custom excerpt imports. If so, then you could customize your WP theme to make use of the special excerpts on the category pages.

For instance, import the products to include the images and links within the excerpt of the entry. Then create a category template that pulls the excerpt info for each post which would then link out to each individual product page.

4. When using DatafeedtoRSS and using the "Import" feature on WordPress is there a way to "schedule" the posts to be random?

Unfortunately no. When WordPress uses the RSS import, it uses the dates within the RSS feed and doesn't play nicely with future dated posts (at the time of this writing).

I hope that gets your questions answered, and if you have any others, you're welcome to ask.

~ Teli

 

[Silencio]

Thanks Linda,

Hi Charlie,

Welcome to 5 Star. I'll poke Teli to be sure she sees this thread as she can give you the most help probably.

Thanks for getting in touch with Teli....

Regarding how often to update. Partly depends on the merchant and how often their stock and/or pricing changes. Even if you don't show prices you want to keep up to date so you don't send a visitor to a product page that's out of stock or re-directs to the homepage.

Ideally you should try to set up a chron job to auto update for you so you don't have to do it manually. I would think you would want to update at least weekly if you do it manually but if it's automatic maybe daily depending on merchant???

I will do more research on my merchants and how often they update their datafeeds. And setting up a cron sounds like a good idea but wouldn't I have to clean and prep my feed first and if I am doing that I would assume that it would need to be updated right away if it was a new updated feed....

I'm not sure about this advice because I don't get as many datafeed questions lately so have not kept up with things as much as I used to on this front.

Now I did spend a little time on this forum before posting I have it from a good source (this forum) that you are pretty much an expert (datafeed queen) and that you give out prizes (crowns) to those that preform highly with datafeeds (datafeed kings).....so I guess that I need to shake the tree of knowledge a little bit....to get you back up to speed

Thanks for your help.

Charlie

 

[Silencio]

Hello Teli,

Actually, I haven't received them. Now this has me worried that I'm not receiving others' e-mails if they use the contact form on the site, because I set it up to always deliver the e-mails to me...I tout it as the most reliable way to reach me (sending a message via the contact form).

If you don't mind, would you help me figure out why your messages aren't coming through? First, let me know which form you're using to try and contact me (direct link is helpful), the e-mail address you're using (feel free to PM me with it so you don't get on any spambot radar), and whether or not you received any errors when submitting the form.

I just emailed you from the forms in the WPDFI and the DF2RSS sites to see if you get them. When I did I received an automated reply, like the one that I got before and I was able to search my emails for "WPDFI" and found that it was indeed that site and that I had emailed you on April 05, 2007 and then again on April 19th, 2007. And I just figured that you are very busy and/or didn't get them.....

But they were questions about datafeeds using the WPDFI program as I was considering purchasing it and wanted to make sure of a few of my questions about it first. Needless to say that since I didn't hear back I was unsure as to if I should proceed. Sorry...

Depends on how often the merchant updates.

I guess I need to try and figure this out and would assume that it would be different for each merchant. I will do some research into this and see what I can figure out.

I think I have heard the term "Live Feeds" that automatically update. That would be really nice to have but I am thinking that this doesn't happen very often, does it?

Unfortunately, I've only used RSS2Blog a few times for other clients so cannot properly answer your question.

Thanks, not a fair question for you to try to answer....thanks anyway for trying.

I can say that WP's built-in RSS import utility will not republish an entry that has the same title. So, if the datafeed has been updated and the title of the product remains the same, WP wouldn't import it again...however, it also wouldn't update the entry with information that's changed either.

Unfortunately, you may need to contact Michelle for an answer to this question. I know on the WPDFI script, there's a setting to "overwrite" the information of previous imports, so there may be something similar in RSS2Blog.

Well, again, I'm not well versed with RSS2Blog. Also, I don't know whether it would support custom excerpt imports. If so, then you could customize your WP theme to make use of the special excerpts on the category pages.

For instance, import the products to include the images and links within the excerpt of the entry. Then create a category template that pulls the excerpt info for each post which would then link out to each individual product page.

I am thinking out loud here and know that there has to be some way to control this especially if sites that have hundreds of datafeeds need to be updated on a frequent basis. I just have to dig deeper.....to find an answer. Thanks.

Unfortunately no. When WordPress uses the RSS import, it uses the dates within the RSS feed and doesn't play nicely with future dated posts (at the time of this writing).

That is what I was thinking, just thought I would confirm it.

I hope that gets your questions answered, and if you have any others, you're welcome to ask.

~ Teli

Thanks so much for taking the time to assist as you could. I do appreciate it.

As important as datafeeds are and seem to be I personally am finding it difficult to find answers to many questions that one might have concerning them. So it seems that I will have a lot of learning curves along this path.....But I am sure that I can find what I need to know somewhere.

Thanks again!

Charlie

 

[TheCaymanHost]

If you haven't done so already (I think there is already a link somewhere on these forums) pay a visit to Carsten Cumbrowski's pages dealing with data feeds.

Lots of useful information and resources to be found there no matter what your level of experience.

Mo

 

[Linda Buquet]

Good one, thanks Mo!

 

[Teli]

I think I have heard the term "Live Feeds" that automatically update. That would be really nice to have but I am thinking that this doesn't happen very often, does it?

While Live Feeds are fun to work with because they're usually updated often, it can be a pain because they're often delivered in XML format (some are JavaScript or PHP, though). Basically just the data and you'd need a way to take that data and display it on your site -- and some are very inflexible in how you can display it on the site.

Because there's no set standard for datafeeds (I personally believe their should be) like there is for, say, RSS, the available datafeeds vary greatly and not all available datafeed tools can handle *all* datafeeds as-is.

If you happen to come across a merchant with a live feed, make sure to ask if they have any website tools along with it. That would cut out the need for additional software.

I just emailed you from the forms in the WPDFI and the DF2RSS sites to see if you get them. … And I just figured that you are very busy and/or didn't get them.....

But they were questions about datafeeds using the WPDFI program as I was considering purchasing it and wanted to make sure of a few of my questions about it first. Needless to say that since I didn't hear back I was unsure as to if I should proceed. Sorry...

No need to apologize.

Thank you for taking the time to send another message through my forms (I received both), but it's odd I didn't receive them before. Even went back to my e-mail for those dates and they weren't there, so something probably happened on my server's end which didn't deliver the message to me. Quite strange and I hope temporary.

As I said in the e-mail, I've white-listed your e-mail address, so you're welcome to send me a note directly in the future. And I'll respond to your e-mails directly.

Carsten Cumbrowski's pages dealing with data feeds.

Nice resource, Mo.

~ Teli

 

[Silencio]

If you haven't done so already (I think there is already a link somewhere on these forums) pay a visit to Carsten Cumbrowski's pages dealing with data feeds.

Lots of useful information and resources to be found there no matter what your level of experience.

Mo

We are having fun now. Thanks now I am getting lots of info.......on datafeeds. I don't know yet if it will answer my questions but it looks like a great resource.

I sure appreciate it!!! Thanks.

Charlie

 

[cumbrowski]

Hey Mo, thanks for pointing to my resources to data feeds.

I read the original post and have to say that my resources will not help Charlie very much with his problems with RSS2Blog and DatafeedtoRSS. You might noticed that I do not refer to those services at all. That is by choice and not by accident because frankly, I fail to see their value.

Just creating blog posts from data feeds sounds to me not much better than somebody who uses blog feeds to republish them without providing additional information or any other type value. I would also be careful with that. Affiliates who remember Google's infamous "Florida Update" in 2003 will know what I am talking about. Affiliates were able to cash in via using data feeds and tools like web-merge to create virtually a new site with hundreds or thousands of pages on a single day. This income dried up virtually overnight, when Google's algorithm update, dubbed "Florida" was making those sites disappear from their index.

I have not seen good examples of the use of the mentioned tools. The examples I saw so far do nothing else than affiliates did up to 2003, with the only difference that they are using blogs today instead of static HTML websites. The obviously missing feature of allowing feed updates without duplication of existing content shows that those tools were not designed for that. I would be happy to get proven wrong by providing examples where those tools are used in a good way and be more efficient doing that compared to other tools out there that do not convert data feeds to RSS.

RSS itself is already limiting and an ill fit for data feeds. Useful information, such as the price, manufacturer or attributes like color and size etc. must be squeezed into the description element of the RSS feed, because no elements exist for those things in RSS.


Working with raw data feeds yourself is not easy. There are several problem with this that make things more complicated than they already are by nature (its an interface and interfaces are not simple, unless they are standardized). For example does no standard exist for feeds. They come in all shapes and sizes. Add to that, that most merchants don't know much about data feeds and make basic errors that causes big problems for you if you don't check the data feed integrity every time you pull them. The networks are in general no different, their data feed technology did not improve much over the last 5 years. If you create something of value with the feeds, you can't get around the need to update them frequently. How frequently that is, really depends on the merchants, some update their products daily, some only once or twice a year and everything in between.

My resources will help you with that.
Affiliate Product Data Feeds Resources, Affiliate Network Services Documentations, Guides and Articles

This page links to the relevant stuff, from general guides to very specific things like network data feed structures and scripts/source code how to pull them into your database for processing.

If you promote individual products and blog about them for example, you will not update them of course, but you also have no need for data feeds to do that.

Because networks and merchants failed both, 3rd party vendors popped up to provide non-tech savvy affiliates with tools that make sense, like pulling individual products out of a product catalog (if you ever tried that at CJ for example, you will know why there was a need for this), but also dynamic product creatives that are simply a few lines of javascript with some parameters you can define to determine the types of products to show and the layout. The actual product data are being pulled from the vendor every time the page on your site gets loaded and the vendor worries about keeping the feeds information of each supported merchant up to date.

You can find a number of services that are available for that here.
I don't have a favorite. Each of them is slightly different than the other. MY advice is to check them all out and pick the one that works best for your needs.

Affiliate Product Data Feeds Resources, Affiliate Network Services Documentations, Guides and Articles

Another option is for developers who do not want to deal with the nightmare of updating data feeds themselves and kicking a huge amount of data around that can cause your site to crash and render unusable if you don't watch what the heck you got send from the merchant with the new feed. I am talking about APIs or Web Services here. Amazon and eBay have APIs for ages, networks more the most part don't. CJ launched their web services beta last fall, which is still in beta today for good reasons. A small network that allows access to product data on the fly via an interface is AvantLink. After that are the solutions, if any exists at all more like a workaround than a real help. This includes real-time data feeds. Some APIs (the web services versions of them) use XML. I have a separate resources page just for web services here.

Web Services Development Resources, Articles, Tutorials and Books for Developers

There are other great tools, services and resources out there that have APIs and are useful in combination with product APIs. You can find a bunch of them at the following page, which also refers to resources where you can find out about the vast number of available APIs from the zillions of Web 2.0 services out there who (for the most part) embrace the use of standardized APIs, like SOAP/WSDL, XML-RPC, SAX and other types of web services.

Website Development Resources, APIs and Web Services, XML, Ajax, .NET, PHP

Okay, Wrapping Up /
Sorry for the long post and my rambling, but I guess it wanted out and ReveNews is still in the process of being upgraded, so no blogging about affiliate stuff for me right now .

I hope that you find the information useful and again, if anybody can point me to examples where those Data feed to RSS and/or to Blog tools where used in a good way that will not be considered spam or "little value adding affiliate sites" and kicked out of search engines and other places eventually, where it isn't already the case today, I am all ear and willing to adjust my attitude towards them.

Cheers and happy Labor Day.

 

[Linda Buquet]

Hi Carsten,

Great to see you again. Thanks SO much for all that info. I can tell you have some pent up posts in you since Revenews has been in the re-design process.

You know whenever you ever feel like talking about industry stuff,
you are always welcome to come visit us and post here!

 

[Larwee]

Carsten, I'm glad you came here and made your post after I notified you about this thread.

It is good to see you here again.

 

[cumbrowski]

A few more comments

I dug up a few more references to some of the important stuff I was talking about.

See: Complete Website Templates for Affiliates.
Complete Website Templates for Affiliates.

Just shows the same problem, with different content, but the concept is the same.

I was not kidding about how bad it was for data feeds affiliates back then. I just found this old post where an affiliate describes his personal experience with the Google update and how it affected him.

Google Florida Dance and the effects on me - ABestWeb Affiliate Marketing Forum

I just came across two interesting articles to "Florida" and affiliates
SEO after Google's Florida Update

and data feeds and PHP/MySQL
Affiliate Data Feeds

They are good read, except for the annoying AdSense ads all over the place, but he seems to have written the stuff himself. It's not free to republish stuff so I can not refer to a different site with the same content (talking about replication). I seriously have to think about linking to him from my site.. mmh.

He mentioned this script, which is sold for $39.95

Datafeed-to-MySQL version 2.11
PHP | mill -- Datafeed-to-MySQL v. 2.11 DEMO

Does anybody uses this and could share his experiences and opinions about it? Is it worth it? I mean there are free scripts out there for this kind of stuff, but they are all over the place and not as flexible as needed to become a one stop shop. The $40 would be worth it, if the script delivers what it promises.
Thanks

p.s. Duplication in itsself is not a BIG problem (its still a problem, but the consequences are different), but massive duplication without providing anything unique is.

 

[cumbrowski]

Hi Carsten,

Great to see you again. Thanks SO much for all that info. I can tell you have some pent up posts in you since Revenews has been in the re-design process.

You know whenever you ever feel like talking about industry stuff,
you are always welcome to come visit us and post here!

Hi Linda,
long time no see, hear, chat, discuss.. whatever hehe

I do have a few pent up posts and ideas flying around, but not in this case. Larwee send me an email and pointed me to the post and I just felt the need to respond to it. I would probably have blogged it and then posted a summary here with a link to it, if ReveNews would have been up

Most posts I do are virtually instantaneous. I am not a news blogger and don't have my own internet marketing blog for that reason. I blog too irregularly.

If the day would just have more than 24 hours, I would come more often here, because I love you guys, but honestly, I think that this would have been better being posted at a blog. It has usually more reach and is easier found down the road to refer to it. I remember my forum posts before I was blogging. Some were actually pretty good (and long) and are now buried between a lot of noise and hard to find. Don't get me wrong, I think a forum is great for things like seeking help and advice for specific problems, but some stuff is better suited for a blog.

Every time a bigger problem came up, did my blog posts turn into essays

I am pretty sure, that you know what I mean, at least I hope that. I don't want to divert from the subject of the thread too much. If you would like to discuss this subject further with me and/or others, feel free to start another tread for it or shoot me an email and we can't talk about this privately.

Cheers!
Carsten

 

[Linda Buquet]

Yes I totally understand what you are saying Carsten. Funny though for lots of reasons I'm starting to shift away from blogging and putting more focus on posting here more. Forums are still a very important part of social networking and so much more interactive than blogging.

But you're right, this thread is about datafeeds.
Thanks again for posting all your valuable insight!

 

[cumbrowski]

Btw. Blogs start getting the same problem . People start using social media platforms more and more for things that used to be blogged. Some even asked the questions if blogs are going to die. Nope, they will probably not anytime soon, but stuff where blogs were not the best choice to do it in a first place, will be replaced by things that are better suited for it and blogs will be used for what they are still best for, just like forums.

So, back to datafeeds.


I did not elaborate on the use of web services and the problems with real time data feeds in my first comment and some people who don't know about those kind of things might have a problem with understanding that part of my comment.

APIs (in form of a web service or not), allows to get real-time data for a small set of products or even specific products. In case of CJ, real-time as in the information CJ has for the product at that moment. If the merchant did not update the feed at the CJ network, the information could still be wrong.

Real-time datafeeds as in pulling the feed with current information (and not from whenever a data feed file was the last time provided) still has the problem that you get too much data back and still have to do the full processing on your end.

Since merchant provide the feed though, is the possibility of errors and problems with the feed somewhat smaller, since the network as middle man who is in some case causing the problem, is out of the loop.

If the real-time interface allows filtering of the results down to a very small set of products or even individual products, we start calling it API and not FEED anymore. A simple filter by department/category or type of product is sometimes an option, but still returns more data than you need. This adds to the overhead for transfering and processing the data and makes it often inefficent to use that kind of interface for real time calls on the fly when a user hits a page on your site. That on the other hand causes issues for you and the merchant if you have a large amount of traffic and calls to make.

I hope that clarifies things a bit more

 

[Silencio]

Carsten,

Thanks for stopping by and sharing about this topic. I appreciate it very much and I am sure that others will too.

While I am just getting my feet wet with Datafeeds I am learning quickly that this powerful media and marketing technique is very lacking in many ways!

I am not a programmer and have only dabbled enough to know that is another profession....that I am not going into, but what I do know about it is enough to know that I need help with it.

While my first attempt at getting datafeeds on my pages indeed did use the program RSS2Blog to post it to a Wordpress Blog this program and Wordpress also can post to regular html pages. I am sure you are aware of that also as Wordpress has really evolved into a powerful CMS software instead of just a Blogging software.

So in my first attempt, with the available information that I had available, was somewhat successful in that I did indeed get datafeeds uploaded to my page.....but that was the start of my questions....that brought me here to ask questions.

While this first attempt I consider a minor major accomplish for me in my goals to learn datafeeds I now realize that this was just a baby step in the right direction.

I have had web pages since 1995 and have been building my own since 1999 and I have made it a big effort to learn as much as I can about SEO so while I would not consider myself an expert I am literate of the topic and can hold my own to a certain degree.

I was also dropped off the charts in the Google Florida Update but it was not because of a duplicate content penalty as all of my content on that main page of mine was original and created by me. I think a lot of other factors were involved. And in today's world there is a lot of duplicate content posted on web pages especially in the form of RSS feeds and it doesn't seem to affect them to the extent that maybe it did, maybe because of the rise of RSS since the Florida Update. Just an observation.

After TheCaymanHost posted your web page (Thanks again) I have spent quite a bit of time there and you have a tremendous amount of information there to digest.....and explore and it looks and feels like I will find lots of answers there but....

in my searches so far I have not found a good guide that says "If you want to get started with Datafeeds this is the best way to it from the Beginning" which is why I have been going around the block in this learning adventure.

What I find is "You could do it this way" or "You could do it that way" or "Maybe this way".........guess I chose a way that might not work as well as others although I have this strong feeling that if a way is found that would allow one to update feeds easily in a WordPress application it could out perform some other options.

I see a huge need for a revolution in the way datafeeds are produced and distributed and I know that you have been dealing with this issue much longer than I. Is there a reason that datafeeds are not more standardized?

Carsten you advice is much appreciated! Thanks. I hope that through your help and resources I can find a good solution or two or three.....

But let me ask the question as I end this post.....

If you, with the knowledge that you have today, were to recommend to someone the best way to get started with Datafeeds, to get them started in the right direction, what would you suggest to them in a step by step approach?

Thanks

Charlie

 

[Silencio]

He mentioned this script, which is sold for $39.95

Datafeed-to-MySQL version 2.11
PHP | mill -- Datafeed-to-MySQL v. 2.11 DEMO

Does anybody uses this and could share his experiences and opinions about it? Is it worth it? I mean there are free scripts out there for this kind of stuff, but they are all over the place and not as flexible as needed to become a one stop shop. The $40 would be worth it, if the script delivers what it promises.
Thanks

Carsten I also found this post on another forum where someone posted a free script that will upload to an MySQL database.


(If this outside link is not allowed to be posted here please remove it and accept my apologies but as were were discussing this I thought it might be of value...)

And I have a question for you on this Datafeed to MySQL approach if you could please help me understand the importance of this technique.

If you have uploaded datafeeds directly to your MySQL database today and next week you need to Update your datafeed, how is this accomplished?

And also in relation to my site since a datafeed in a WordPress site is also input into a MySQL database is there an easy way to update a feed by uploading directly to your MySQl database?

Charlie

 

[cumbrowski]

Hi Charlie,

And in today's world there is a lot of duplicate content posted on web pages especially in the form of RSS feeds and it doesn't seem to affect them to the extent that maybe it did, maybe because of the rise of RSS since the Florida Update. Just an observation.

If you are interested in the whole duplicate content thingy, have a look here. There is no simple answer to that one either .

Duplicate Content and Near Duplicate Content Issues -Resources for SEO

in my searches so far I have not found a good guide that says "If you want to get started with Datafeeds this is the best way to it from the Beginning" which is why I have been going around the block in this learning adventure.

The thing is that the subject is for beginners and requires serious technical knowledge to do it right. There are some scripts but they either produce something that is pretty much worthless or you are required to customize it so extensively that it is not easy again. But I see your point, its not spelled out very often. I am thinking about taking some of my responses here and craft something out of it to put on my site.

I see a huge need for a revolution in the way datafeeds are produced and distributed and I know that you have been dealing with this issue much longer than I.

Amen to that. I can point you to forum posts that are 5 years old where you can hear me ***** and yell (after I made some very specific and relevant suggestions). Regarding the why. Not sure, not a priority with the networks, obviously.

Carsten you advice is much appreciated! Thanks. I hope that through your help and resources I can find a good solution or two or three.....

My pleasure, I am glad that I could help.

If you, with the knowledge that you have today, were to recommend to someone the best way to get started with Datafeeds, to get them started in the right direction, what would you suggest to them in a step by step approach?

- Get at least some programming skills and also some experience with database software (MySQL or MS SQL Server (which I use))
- Get familiar with delimited text files and their problems. You can use Excels import feature to import feeds. There you will see fairly quickly what some of the issues are
- never assume that the feed file is okay and that the columns are used for what they are supposed to be used
- Get familiar with FTP (basic FTP that comes with Windows is all you need)
- Get familiar with XMLHTTP to pull data via HTTP
- Get familiar with data validation and data normalization principles and problems
- Learn how to schedule things for automation. I use SQL Server Agent to schedule things, but the Windows scheduler would work for a lot of things too
- Basic XML knowledge would not hurt either
- Get familiar with web services and use them whenever you can.
- Amazon's and eBay's APIs are good for learning and getting experiences with web services


If this is beyond you, have a closer look at the 3rd party services that do all this work for you and provide product data in an easy to use format.

Carsten I also found this post on another forum where someone posted a free script that will upload to an MySQL database.

That one is for the ShareASale data feeds and not generic, although it could probably be modified to use it for other feeds as well, but you need some PHP knowledge to be able to do that.

(If this outside link is not allowed to be posted here please remove it and accept my apologies but as were were discussing this I thought it might be of value...)

It is okay, because it is relevant to the discussion

If you have uploaded datafeeds directly to your MySQL database today and next week you need to Update your datafeed, how is this accomplished?

Two options

You receive in almost every case a full product feed.

1) Delete all records related to the feed in the database and then import it as if it was new. Depending on the feed is the key for the deletion filter usually the merchant ID on the network where you pull the feed from. The problem is that you can't use your own primary key for products in this case, because the key values would change and all your references to the old records would become invalid

2) Check if the product exists in the DB by doing a look-up with a unique identifier for the Merchant+Product combination and do an update if it exists (have a date stamp for "last updated" and update that as well). Insert all records you can't find in the DB. Delete all records where the "last updated" stamp was not updated. This requires more logic for the import script and takes longer to import data.

Hope that helps
Cheers

 

[TheCaymanHost]

Phew - I'm certainly glad I pulled Carsten in on that one

I realized that the original question was not really answered at your site, but you still have one of the most comprehensive and understandable resources out there on the whole subject.

Sorry I created so much work for you! Thanks for helping out, and clarifying so much in your posts - I'm sure this thread will now be a great help to many forum visitors.

Mo