Re: Need Estimate of Programming Effort

"JS" == Jeff Sheffel <bogus@xxxxxxxxx> writes:

JS> I'm looking for a simple estimate of a "level of effort", for a Perl
JS> programming task.

you need to hire someone just for this task alone.

JS> The estimate should be in hours. (Maybe a range of programming hours -
JS> based on level of Perl experience.) Any other additional estimate
JS> information, and design comments are appreciated. (Please do not ask
JS> questions about the requirements, since I did not write them; make any
JS> assumptions necessary.)

you can't do that. i have written crawlers before and the client will
ALWAYS make many changes as it is developed. these projects cannot be
properly estimated without a very clear and precise spec. you are asking
for a world of trouble otherwise and this comes from deep experience.

JS> Design a web scraping utility to scrap information from various
JS> shopping sites. The code should be written in Object Oriented
JS> Perl, with use strict and warnings enabled.

oh boy! strict and warnings add many hours to any project. a stupid
requirement which doesn't help at all with estimates. so many design
questions will need to be asked and answered. this is not a toy.

JS> The initial sites used should be http://www.shopzilla.com and
JS> http://www.shopping.com. However, the program should be designed in a
JS> manor that will allow other sites to be added in the future.

s/manor/manner/

and you can't crawl those sites as is. they are shopping search engines
so you would need to know the product names/etc to locate them.

JS> The minimum requirement for output is: Site scraped from, product name,
JS> short description, low price, high price. For simplicity scrapings can be
JS> limited to 60 items or less from each target site.

which 60 items? is there a list? will it grow? more unasked questions.

JS> Optional features that can be added are throttling and threading.
JS> Throttling will limit the number of hits to a particular site in a
JS> giving time period and threading would allow the program to make
JS> several requests simultaneously.

parallel requests can be done without threading and in several
ways. threading is a design issue and not a requirement. throttling is a
requirement and if you didn't do it, any decent site will notice and
block you. are these 'optional' features to be designed in now or bolted
on (poorly) later? again, crawling large scale is not for
kiddies. prototype crawlers will not scale unless they are designed for
it from the beginning. so you have a major requirements conflict here
about whether this is a kiddie toy or a professional scalable crawler.

JS> The program should be fully documented and run without warnings.

they want documentation too? unheard of!! how about some properly
written requirements first?

if you really need professional help (and i think you do) have them
contact me directly as i have actually created 2 major crawler systems
and can at least ask the right questions. but there is no way in hell i
would provide a time estimate on such a frivolous set of
requirements. you can't make assumptions as this could be a week long
kiddie thing or 6-12 man-months which is a pretty wide range of
estimates.

i await the call from your client (or yourself). (not holding my
breath).

uri

--
Uri Guttman ------ uri@xxxxxxxxxxxxxxx -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
.

Relevant Pages

  • Re: Need Estimate of Programming Effort
    ... So, creative and rapid programming disciplines are being called for, here. ... and design comments are appreciated. ... i have written crawlers before and the client will ... JS> Optional features that can be added are throttling and threading. ...
    (comp.lang.perl.misc)
  • Need Estimate of Programming Effort
    ... I'm looking for a simple estimate of a "level of effort", for a Perl ... Design a web scraping utility to scrap information from various shopping ... Optional features that can be added are throttling and threading. ...
    (comp.lang.perl.misc)
  • Re: passing data
    ... >>> running script in the first place. ... > your design first before you start coding with only half-formed ideas. ... Ok Matt I take your point but in this particular instance I must admit that ... Ok I will confess, it's just a means to get used to coding in perl, I have ...
    (comp.lang.perl.misc)
  • Re: [SLE] Programming environment selection.
    ... It's part of the PerlQt package. ... > about everything I've ever needed in Tcl or Perl. ... when you use OO design the complex data structure ... better than hashes of hashes, arrays of arrays of hashes etc. etc.? ...
    (SuSE)
  • Re: Perl Performance Tuning
    ... o> This should be required reading for all Perl hackers, ... who in their right mind uses lots of bit flags for booleans in perl? ... use undef instead of 0 makes no sense. ... fix bugs when they were used outside their very tiny design envelope. ...
    (comp.lang.perl.misc)
Loading