Log In
Home
(1 votes) 12:10 am
June 28, 2022
Offline
My syndication log shows that mySQL spends about 20 seconds looking for duplicates. With 150,000 records in the posts table and a low end server, I guess it's normal. Some of the syndicated RSS feeds have 50 items or more. So, if the "Maximum number of posts" is not reached, syndication time is 50 x 20 s = 1000 s (processing post and executing user-defined PHP code is done instantly).
Is there a way to limit the number of item processed in a feed? If at most 10 items were processed, the syndication time would be reduced to 10 x 20=200 s.
12:17 am
July 2, 2009
Offline
Say your feed has 20 items. You limit the plugin with first 10. That means it will never import all 20? Perhaps in your particular case it's ok, to be able to process only 10 first posts and forget about the rest 10 ones, but 99% of CyberSEO Pro users won't like it and they will be very confused with such an option. I'll be getting 100 emails a day like: "the plugin stopped working!"
1:28 am
June 28, 2022
Offline
I'm getting news feeds. Depending on the quantity of daily news, the feeds are syndicated every day or every hour so that I never need to read more than 10 news items in a feed. I'm surprised only 1% of users are in this case!
So my only option is to improve mySQL response time?
Thanks for your time. I love CyberSEO and its possibility to add custom PHP code!
6:05 am
December 17, 2022
Offline
I avoid this issue by having more frequent checks with a smaller number of items. I know that in my case most sites won't add more than 5 posts in a couple of hours, so it is safe to check for the top 5-10 every hour or 2 rather than try to get all 20-30 at the end of the day. It also keeps the site fresh all the time with new content. Of course some feeds only update one post a day, so this has a different frequency.
Also the update that removed the Full Text fetch before the duplicate check has helped make the script faster.
9:09 am
July 2, 2009
Offline
joel.nusbaumer.evearly said
So my only option is to improve mySQL response time?
Actually my answer will be very trivial. The only way to do that is to upgrade your version of MySQL and/or upgrade the server hardware. The request itself is very simply.
Also you can alter 
Login to see this link. In most cases the duplicate check by GUID only will be enough. If you don't modify the post titles, you can check for duplicates by titles only, which works even faster, because it uses the most simplest request.
P.S. I really can add such an option to limit the number of unsuccessfully tires. But you can't even imagine how many CyberSEO Pro users don't read the documentations at all, but they just like to change all the settings just because... they see them on the feed settings screen. So I'm getting a lot of emails that say: "the plugin stopped working! fix it immediately!", and every single time that happens because they selected a wrong option and because they just want to check and analyze the Syndicator log :(
10:52 pm
June 28, 2022
Offline
Login to see the quote
Yes the duplicate search is much faster if GUID only is selected. And yes again, GUID only can be used for most of my feeds.
I just realised that the new "Preprocess source PHP code" will allow me to modify the feed and delete items if they are in excess of 10.
Using these 2 solutions I expect to reduce the processing time by 10 to 50 times depending on feeds.
So, case solved, no need to migrate to another server yet.
Thanks!
11:27 pm
July 2, 2009
Offline
Yes, you can put a simple PHP code into the "Preprocess source PHP code" box like this:
Login to see the code
It will limit the syndicating feed by the five first items.
All RSSMost Users Ever Online: 541
Currently Online:
10 Guest(s)
Currently Browsing this Page:
1 Guest(s)
Top Posters:
ninja321: 84
s.baryshev.aoasp: 68
Freedom: 61
Pandermos: 54
MediFormatica: 49
B8europe: 48
Member Stats:
Guest Posters: 337
Members: 2853
Moderators: 0
Admins: 1
Forum Stats:
Groups: 1
Forums: 5
Topics: 1640
Posts: 8352
Newest Members:
wanmarkets, torontomark48, info.ckmedianetwork, contact.mybeautystar, samuelbodde, john.prushAdministrators: CyberSEO: 3947