Journal Search Engine
Search Advanced Search Adode Reader(link)
Download PDF Export Citaion korean bibliography PMC previewer
ISSN : 1598-7248 (Print)
ISSN : 2234-6473 (Online)
Industrial Engineering & Management Systems Vol.11 No.3 pp.266-275

Crowd Participation Pattern in the Phases of a Product Development Process that Utilizes Crowdsourcing

Joonyoung Park*, Anhtuan Tran, Shoaib ul Hasan
Department of Industrial and Systems Engineering, Dongguk University
(Received: December 7, 2011 / Revised: May 22/July 26, 2012 / Accepted: August 1, 2012)


The rise of crowdsourcing and web 2.0 opens plenty of opportunities for companies who want to exploit external sources of ideas for internal innovation. Utilization of crowdsourcing for product design and development has been attracting the attention of both enterprises and researchers. Many cases of implementation of crowdsourcing for product design and development such as: Threadless, FIAT Mio, Lego online Factory, etc., have made crowdsourcing a promising alternative source of innovative power. Although crowdsourcing gained access due to improved Internet access and web 2.0, it is little understood how the crowd, with respect to participation, behaves for any crowdsourcing project. To investigate this, we conducted an experiment on a real crowd of engineering related individuals to figure out the crowd participation pattern for various product design and development phases of a new product development project. The experiment results give a quantitative view of the participation of the crowd (i.e., crowd participation pattern) in various phases of a product design and development process that utilizes crowdsourcing, provide a practical guidance for companies to harness the power of the crowd sensibly, and provide experimental data for further research in this field.

11-3-6_266-275_ Joonyoung Park.pdf407.5KB


1.1 New Product Development

According to Ulrich and Eppinger (2008), new pro-duct design and development (NPD) is defined generally as “the set of activities beginning with the perception of a market opportunity and ending in the production, sale and delivery of a product.” Market opportunity may come from market pull or from technology push. The “technology push” describes a situation where an emer-ging technology or a new combination of existing technologies provide the driving force for an innovative pro-duct and problem solution in the marketplace (Herstatt and Lettl, 2000). Conversely, the term “market pull” implies that the product or process innovation has its origins in latent, unsatisfied customer needs in the marketplace. The identification of these needs occurs first and is then followed by the required development activities (Chidamber and Kon, 1993). There are two parallel paths involved in the NPD process: one involves the generation of ideas (idea generation), development of concepts (concept development), and detailed engineering calculations (detailed engineering); the other involves market research and marketing analysis (market pull) or internal R&D (technology push). Engineering oriented researchers are interested in the first path which is com-mon in both technology push and market pull strategy; it consists of idea generation, concept development and de-tailed engineering. Typically, the steps of an NPD project are shown in Figure 1.

Figure 1. Typical steps in a new product design and development (NPD) project.

NPD plays an important role among the activities of a company. Companies typically see new product development as the first stage in generating and commercializing new products within the overall strategic process of product life cycle management used to maintain or grow their market share. Conventionally, NPD is executed inside the company within the NPD team which consists of engineers and experts with various majors: mechanical engineering, electrical engineering, industrial engineering, finance, marketing, etc. The external resour-ces are limited to close partners or vendors.

1.2 Crowdsourcing and Its Application to New Product Development

In the mid 2000s, the Internet explosion and Web 2.0 era opened opportunities for a boundless collaboration in NPD. Plenty of companies executed their NPD projects not only in the office but also on the open Internet environment. By making open calls to the “crowd” on the Internet to attract their attention to NPD problems, some companies can grasp solutions and suggestions for their NPD projects in short time at low cost (Brabham, 2008; Schenk and Guittard, 2009; Fahling et al., 2011). As a case of inviting the crowd to an NPD project, FIAT Brazil-an automobile company made an open call for designs and suggestions for their new model named Fiat Mio CC. The Fiat Mio CC project website at draws together 14,000 visitors from 140 countries, most of whom can be considered as potential customers (Brondoni, 2010). 

The action of giving an open call to the “crowd” on the Internet and getting feedbacks in the form of solutions or suggestions is termed “crowdsourcing” by Howe (2006) in an article in Wired Magazine. The term “crowd-sourcing” is the combination of “crowd” and “outsourcing”. Howe (2009) defined “crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” 

 Crowdsourcing is based on a simple concept: anyone has the potential to plug in valuable information (Greengard, 2011). (Howe, 2006; Piller, 2010) is a website that exploits crowdsourcing for general problem solving. In this case, companies who have unsolved engineering and scientific problems can post their problem on InnoCentive website and receive solutions submitted by individuals from the crowd. Once a solution is selected to be used, the company will patent it and give the reward to the solution provider. 

There are plenty of companies applying the crowdsourcing concept for product development. In Figure 2, Kim et al. (2010) introduce a classification result of cases of applying the crowdsourcing concept for NPD.

Figure 2. Classification result of cases of applying the crowdsourcing concept for new product development.

1.3 Motivations for Crowdsourcing

There are several reasons that can explain why individuals are willing to join a crowdsourcing project.
Most important reasons might be the following:

1.3.1 Financial rewards

In most crowdsourcing projects, individuals from the crowd are encouraged to join the project to gain financial rewards. For example, in Mechanical Turk (Howe, 2006; Horton and Chilton, 2010)-a website belonging to Amazon, individuals can be rewarded as they complete a certain task (termed human intelligence task [HIT]) required by the company (on behalf of their clients), or in the case of, the winning solution provider is the only person who will be rewarded. The amount of financial rewards may vary from tens of cents for each task in the case of Mechanical Turk or hundreds of thousand US dollars in the case of big projects posted on Doing some jobs in free time with no string attached and having chances to earn extra money is an important motivation for the participation of the crowd in a crowdsourcing project.

1.3.2 Reputation and recognition

Besides crowdsourcing projects that offer financial rewards, there are many crowdsourcing projects which do not offer any kind of payment for the participants. But still, there are plenty of individuals joining those projects for free. In the case of Yahoo! Answers (Wightman, 2010), a website where users ask and answer one another's questions, a user who provides good answers will gain points, and the higher the points he has, the higher the reliability his answer will be. The points here are considered as the user's reputation. 

1.3.3 Opportunities

By joining a crowdsourcing project, especially in community websites, participants can gain many kinds of opportunities. The crowdsourcing project itself is a place where participants can show off their best and get attention from potential customers, partners, or recruiters.

In the case of Flickr (Kleemann et al., 2008), individuals are allowed to upload their photos on the website. Some people upload photos for fun while others use the website as the place for their free online galleries to show their best artworks to the world, which includes people who potentially want to purchase the works, hire the author or make some kinds of co-operation. 

1.3.4 Joy and fun

Some persons join a certain crowdsourcing project for nothing but joy and fun. Facebook-the largest social network service so far can be referred as an example. Most of the Facebook users use the website to share their status, thoughts, stuffs, etc., and connect to their friends, colleagues, family to update information. 

1.3.5 Contribution willingness

 Wikipedia (Kleemann et al., 2008) is a website that allows anyone to submit, edit, and discuss on articles about any kind of topics. The quality of the contents posted on Wikipedia is surprisingly high and the motivation of the individuals to continuously update articles is their willingness to contribute.

1.3.6 “Prosumer” trend

 “Prosumer” is the combination of “pro” and “consumer.” Consumers nowadays are no longer satisfied with the pre-configured products and services offered by manufacturers. They want to join the “co-creation” pro-cess to put their own ideas on the future products and services.

Threadless (Howe, 2009), Fiat Mio CC, Dell Idea Storm (Kleemann et al., 2008) are representatives of crowdsourcing websites that fit this trend. Voices of customers are heard and good ideas are grasped to make products and services that people are willing to buy. 

Table 1. Major motivations for crowd participation

The major motivations for crowd participation in a crowdsourcing project are listed in Table 1. 

1.4 Web 2.0-Technical Prerequisites for Crowdsourcing

 It is the Web 2.0 which allows the crowd to participate in a crowdsourcing project. Web 2.0 has interactive forms that allow individuals from the crowd easily to participate in the project by submitting content such as: ideas, suggestions, files, etc., and leaving comments, votes, etc. Web 2.0 is the environment for users to generate content (termed “user generated content”) (Kleemann et al., 2008). Without these features of Web 2.0 (or later web generation), crowdsourcing projects cannot be executed on the Internet.

1.5 Crowd Participation over Phases of a Crowdsourced NPD Project

 As the name suggests, in a crowdsourcing model, crowd is the core factor. Therefore, the crowd participation pattern (for a crowdsourced NPD project) must be analyzed for an effective implementation of crowdsour-cing. Generally, most of the previous research mentioned reward as a factor leading to participation of the crowd in a crowdsourcing project (Ebner et al., 2009; Albors et al., 2008; Horton and Chilton, 2010). In a crowdsourced NPD process, other factors such as: NPD phase characteristics (“process” factor), product type and complexity (“product” factor), as well as crowd type and size (“crowd” factor) might also have influences on the participation of the crowd.

 For the process factor, each phase in an NPD process has its own characteristics that are different from the others.' Participation of the crowd on different phases is different.

 In the work of Kim et al. (2010), the number of cases for different phases is different. In this work, among the total of 33 cases surveyed, 13 of them (i.e., 39.4%) laid on the phases of Concept/Design; while only 6 of them (i.e., 18.2%) laid on the phase of Testing/Custo-mer's Rate (Figure 2).

There have been many cases of applying the crowd-sourcing concept for new product development but the surveyed cases are relatively noticeable ones in terms of attracting crowd participation. Hence, the distribution of cases through phases might imply the dependence of crowd participation on phase characteristics. This paper aims to investigate that kind of “crowd participation pattern” over NPD phases. 


2.1 Open Innovation

Ward et al. (1995) suggested that sets of design alternatives work better for Toyota designers to reach the final answers than pursuing one alternative iteratively. Open innovation can bring chances to obtain design al-ternatives. Dunphy et al. (1996) argued that there are 7 macro and micro level discriminators making technological innovation happen: 1) the technological prerequisites; 2) the country's socio-cultural tendencies; 3) the country's material, human, and institutional infrastructures; 4) the structure of the particular industry; 5) the size and nature of the individual firm; 6) the attitudes of the management of the firm; and 7) the establishment of standards for the widespread diffusion of the innovation. These 7 discriminators apply for open innovation as well. 

 Chesbrough (2006) pointed out that, in today's distributed knowledge environment, a company can hardly afford to rely entirely on their own ideas to advance their business. Open innovation enables companies to create and profit from their own ideas as well as others' ideas. Open innovation outlines a new environment for R&D, and demonstrates that this new environment replaces the logic of an earlier era, where innovation is closed from outside ideas and technologies.

West and Gallagher (2006) suggested that user involvement can help to obtain innovation by allowing users to contribute to the development of products using pooled product development strategy. 

 Open innovation can help to improve new product development by suggesting new ideas and design alternatives. Open innovation is sometimes referred to contain crowdsourcing as one of its forms (Ebner et al., 2009; Bartl et al., 2010). Some authors mentioned crowdsour-cing as “community-based innovation” (Elmquist et al., 2009). Schenk and Guittard, (2009), on the other hand, argued that open innovation and crowdsourcing have differences in their targets (“innovation” for open innovation), and interacting objects (“firms” for open innovation).

 Dekkers (2011) discussed Living Labs as innovation networks, the position of Living Labs as a part of the innovation process, the move from closed innovation to open innovation and the necessity to manage innovation within the setting of networks to collaboratively deliver new products and services to potential users. He implied that open innovation aims at increasing the throughput of innovation processes and at reducing uncertainty through interaction with users during new pro-duct (and service) development.

2.2 Application of Crowdsourcing to New Product Development

Crowdsourcing for new product development has gained popularity recently. Kleemann et al. (2008) reviewed phenomena related to crowdsourcing, factors contributing to the increasing prevalence of crowdsourcing and suggested 9 types of crowdsourcing by the form they apply. Among these types, 'Participation of consumers in product development and configuration' and 'Product design' can be directly applied to NPD. 

 Hinchcliffe (2007) claimed that “leveraging crowdsourcing effectively” is a critical factor for “product de-velopment 2.0” in terms of competitive advantage. In other work, Hinchcliffe (2009) suggested five functional business areas that are suitable for applying crowdsourcing. These are problem solving, design, collaborative work, testing and support. Furthermore, Bertoni and Chi-rumalla, (2011) suggested that crowdsourcing, through Web 2.0, can be an intuitive way to leverage bottom-up tools for the benefit of product development where customers and the crowd play in the innovation process. Snow et al. (2011) also agreed that crowdsourcing can be used as a problem solving model for collaboration issues in organizing continuous product development and commercialization. One of the reasons that crowdsourcing can be a source of product innovation is that there might be someone outside the company who knows the solution to the problems faced inside the company (Panchal and Fathianathan, 2008). Poetz and Schreier (2012) claimed that crowdsourcing initiatives among users can actually outperform professional in-house activities for the generation of product ideas, at least under certain conditions. Taha et al. (2011) studied user's involvement in NPD and identified factors governing user involvement and designer practices in NPD.

Crowdsourcing for new product development can benefit firms by using consumers' expertise, and the innovation provided by consumers can also be used for marketing purposes (Kleemann et al., 2008; Whitla, 2009). 

Cooper and Edgett (2008) commented that crowdsourcing results are weak (in some cases), and recommend to implement crowdsourcing in certain types of firms. 

Still, there is a lack of practical guideline for firms to decide what kind of task should be outsourced to the crowd in order to gain high rate of participation. 

2.3 Crowd Participation in a New Product Devel-opment Project

 To get good results from crowdsourcing implementation for NPD projects, crowd participation is an important issue. The high crowd participation rate might lead to high chance of bringing innovation (Redlich et al., 2008).

The participation of the crowd in a certain crowdsourced NPD project depends on many factors, such as: reward factor, process factor, product factor, and crowd factor. So far, most of the research considered reward as the major factor that leads to the participation of the crowd to a certain crowdsourcing project. Reward factor is mentioned and considered as the main motivation for crowd participation by many authors (Albors et al., 2008; Brabham, 2008; Malone et al., 2009; Ebner et al., 2009; Fahling et al., 2011). 

Mason and Watts, (2009) claimed that increased payments increases the quantity of work performed, but not its quality, and particular design of the compensation scheme can have a significant effect on quality in a crowdsourcing project. Horton and Chilton (2010) presented a model of workers supplying labor to paid crowd-sourcing projects. In this work, a method of estimating a worker's reservation wage (the lowest wage a worker is willing to accept for a task) has been introduced. Lei-meister et al. (2009) described how activation enabling functionalities can be systematically implemented in an IT-based ideas competition and proposed an approach to designing components for participation support by utilizing incentives and motives of users. Borst (2010) explored the effects of motivations and rewards on participation as well as performance in voluntary online activities. 

However, the effects of other factors (rather than reward) on crowd participation have not been investigated extensively. Only a few noticeable papers mentioned the effect of task complexity which includes the characteristics of NPD phases on the continuance of the crowd's attention to the problem and the crowd's sustained participation (Sun et al., 2012). 

2.4 Research Objectives

Although crowdsourcing gained popularity due to improved Internet access and web 2.0, it is little understood: 1) What are the different factors that affect the crowd participation in an NPD project?; 2) What is the crowd participation pattern for various product design and development phases for a new product development project?; and 3) How can crowdsourcing contribute with respect to innovation in new product development? 

 This paper focuses on the second point and the authors try to answer the question through literature and experiment. The third point is important with respect to crowdsourcing and innovation in NPD. It is a question that calls for another independent research and is not included in the scope of this paper. Specifically, this paper focuses on quantitatively figuring out the variation of participation of the crowd on various phases of a certain NPD process through experimental study.

In this paper we will consider the relationship:
CP = f1 (process|fixed {crowd, product, and reward})
where, CP stands for crowd participation.


3.1 Experiment Proposal for Process Factor

 We performed an experiment on a “crowd” of engineering related individuals in an online engineering community. This community was founded in 2006 and is currently running as an online forum. Most of its mem-bers are engineers, experts, students, lecturers, professors and engineering managers living all around the world and its contents can be accessed worldwide for free.

Table 2 shows some major statistical data about this community website as of March 29, 2011. 

Table 2. Statistical data of the “crowd” environment

In each phase of idea generation, product concept design, detailed engineering calculation, physical prototyping, and design evaluation, we investigated how the crowd responds to the open call. In each phase, we observed how many percent among the announced individuals would download and look deeply at the problem description sheet and how many percent among them would try to solve and submit the solution. By analyzing the experimental results, conclusions are drawn about the participation of the crowd to an NPD problem as well as the “participation pattern” that illustrates different responses of the crowd to open calls for different NPD phases. 

3.2 Experimental Design

The experiment, which lasted for 75 days, was performed in November and December 2010 in the form of a product design and development contest named “NPD 11.” The data from the contest is collected as experiment data and those data will be used for analysis. 

The contest was conducted in 5 phases as shown in Figure 3. 

Figure 3. Timeline of the new product design and development (NPD) 11 contest on the website.

All individuals from the website are allowed to participate in this contest without any restriction. For judging the submissions of participants, the authors establish a committee of experts consists of 7 members who are well known and respected among the community. They are currently working as designers, CAD operators, and engineering managers. For each phase, the organizer of the contest (named “the organizer” from now on) uploads the announcement as well as the attachment of problem description sheet on the website and send invitation to all members to join the contest. Participants are allowed to raise any kind of questions about the contest. Participants submit their solutions via email to the organizer. Once the submission is received from a participant, it will be checked and forwarded to the committee of experts for judging. Judging results then will be sent back to the organizer to announce to participants as well as to identify and reward the winners. 

Figure 4 shows the workflow of one phase of NPD11. 

Figure 4. Workflow of one phase of the new product design and development (NPD)11 contest.

Details about each phase as follows: 

Phase 1: Product idea generation

 In this phase, the participants are allowed to freely propose any product idea that they think to be novel, useful, unique, etc. The proposal should include: name, description, working environment, social contribution, technical and economic feasibility, ease of implementation, environment friendliness, competitiveness, and fur-ther development in the future of the product. Participants are given exactly 10 days for completing and submitting their solutions.

Phase 2: Concept design 

In this phase, the organizer gives participants infor-mation about one product chosen among the proposed ones. The information includes: name, description, work-ing environment, general technical and economic requi-rements. Participants are asked to submit a concept design for that product. The submission should include: 3D images or sketches from different viewing angles, 2D projections, cross-sectioned drawings, explanation of its working mechanism, advantages and optimization of the design and creative product architecture. Participants are given exactly 10 days for completing and submitting their solutions. 

Phase 3: Detailed engineering calculation 

 In this phase, the organizer gives participants a designed concept and its working conditions as well as suggestions of materials range, loading factors, etc. Participants then conduct detailed engineering calculations (mechanical engineering, electrical engineering, materials engineering, etc.) to figure out the specifications of the product (working parameters and product dimensions). The submission should include: 3D view, 2D projections, and cross-sectioned drawings with dimensions, explanation of calculation methods and detail calculation steps, detail drawings for manufacturing of product components (with tolerance) and the choice of materials. Participants are given exactly 10 days for completing and submitting their solutions.

Phase 4: Physical prototyping 

 In this phase, participants are given one product with detailed drawings with dimensions and materials as the result of phase 3. Participants can use existing materials (carton, plastic, rubber, wood, steel, aluminum, etc.) to build physical prototypes of that product. The submission should include: real photos of the prototype with different viewing angles, working videos (if available) and bill of materials (BOM). Participants are given exactly 10 days for completing and submitting their solutions.

 Phase 5: Design evaluation

 In this phase, participants are given a designed pro-duct with 3D images, 2D projections, and cross-sectioned drawings with detail dimensions and photos of physical prototype. Participants are asked to submit their comments and feedback and also suggestions about the design product, its design advantages and disadvantages so that the design team can make corrections, refinements and improvements for final ramp-up. The submitted solu-tion should cover those kinds of information. Participants are given exactly 10 days for completing and submitting their solutions.

The reason why we give 10 days to users in all phases is that we want to make the “same condition” of the crowd, product, reward and also the time to fairly compare the crowd participation for different phases. As we observed from the experiment, for the individuals who submit, they only need at most 4 days to finish the task, even for the hardest tasks (the solutions are submitted within 4 days from the day when the open call is made). That's why we believe that “10 days for all phases” works fine for the tasks in this experiment. Thro-ugh a quick survey, those who submitted as well as those who did not submit agreed that the period of 10 days is sufficient for finishing the tasks. The reason why some individuals did not work on the problem and submit solutions might be that they compared the reward they might get and the efforts they need to make. 


4.1 Collected Data

 Crowd actions in the NPD11 contest are shown in Figure 5 below.

Figure 5. Crowd actions in the new product design and development (NPD) 11 contest.

For each phase of the NPD11 contest, the authors record data about number of views of contest webpage (V), number of downloads of problem description sheet (D), and number of submitted solutions (S). Values of V, D, and S can be achieved from the web and email systems. Results are shown in Table 3. 

Table 3. Recorded data from the new product design and development (NPD) 11 contest

 The data in Table 3 above can be represented in Figure 6

Figure 6. Representation of recorded data in terms of number of submissions in new product design and development (NPD)11 contest.

For a fair comparison of crowd participation among phases, the authors will use ratios instead of original recorded data. The ratios of D/V, S/D, and S/V are calculated. Calculated results are shown in Table 4 below. 

Table 4. Calculated results of ratios of D/V, S/D, S/V for phases

4.2 Data Validation

In this section, we conducted a sample size calculation to prove that the sample in the experiment is large enough to give statistically significant output data. 

4.2.1 Statistics formula for minimum sample size

The minimum sample size needed for statistically significant output data can be calculated as follows:

where SS is the minimum sample size (for infinite population), Z is Z value, which can be retrieved from Z value charts (e.g., 1.96 for 95% and 2.33 for 99% of confidence level), p is the percentage picking a choice expressed as decimal (for example, p = 0.5), and c is con-fidence interval expressed as decimal (e.g., 0.05 = ±5) 

In this paper's case which is provided with sample sizes, p-values, and certain confidence levels, we will consider the confidence interval of the collected data. 

 After applying the above formula for confidence level of 90% (usually used by the researchers), i.e., Z = 1.96, we have the following values of c and possible ranges of the S/V ratios for 5 phases as listed in Table 5 below.

Table 5. Values of c and possible ranges of the S/V (%) ratios

Considering the weak coverage of the ranges of the S/V ratios in Table 5, we can say that with the confidence level of 90%, the data of S/V ratios are statistically significant and can be compared with ensured accuracy. 

Figure 7. Representation of the ranges of the S/V ratios for new product design and development phases. V: number of views of contest webpage, S: number of submitted solutions.

4.3 Discussions

The D/V ratio lets us know how many among a cer-tain number of individuals who are informed about the open call might be interested in the problem and willing to work further to solve the problem. The D/V ratio reflects the level of interest of the crowd in the announced problem due to the degree of complexity and the interest of the problem itself. 

 The experimental data show that the D/V ratio decreases when moving from phase 1 to phase 4 and increases again while moving to phase 5. This means that phases 2, 3 and 4, which require more detailed and skil-led works, are less “attractive” in the eyes of the individuals from the crowd.

The S/D ratio lets us know how many among the individuals who are interested in the problem can solve the problem and submit solutions. The S/D ratio reflects the ability of the crowd to respond to the open call, and once again, this ability depends on the degree of complexity of the problem itself. The S/D ratio has the same pattern of variations over the phases like the D/V ratio. 

Figure 8. Variation of D/V, S/D ratios over phases. V: num-ber of views of contest webpage, D: number of down-loads of problem description sheet, S: num-ber of submitted solutions.

Figure 9. Variation of S/V ratio over phases. V: number of views of contest webpage, S: number of submitted solutions.

The S/V ratio reflects the total effectiveness of the crowdsourcing process in terms of participation. By looking at the S/V ratio, companies can assess how many submissions they can receive for a certain population of the crowd that they can reach through the Internet environment. The S/V ratio shows a pattern similar to the S/D and D/V ratios over phases. This result fits well with the qualitative argument made by Sun et al. (2012) about the relationship between sustained participation of the crowd and task complexity in virtual communities. 

Besides rewards, another factor that drives the participation behavior of the crowd in a certain crowdsourced NPD process is the characteristic of the phase itself. The phases with lower levels of complexity that require less efforts, skills, knowledge, etc., would attract more participants than the phases with higher levels of complexity that require more efforts, skills, knowledge, etc. From Table 4, we have Figures 8 and 9 to show how those ratios vary over phases. By looking at the charts in Figures 8 and 9, we can see that the D/V, S/D, and S/V ratios have the same pattern. They all decrease when moving from phase 1 to phase 4 and increases again while moving to phase 5. 

On the website of Dell Idea Storm, a case that is mentioned by Kleemann et al. (2008) within the 52 newest submissions in October 2011, there are 32 submissions for idea generation, 8 submissions that suggest product concepts, 12 submissions for evaluating Dell's designs; and there is neither submission of detailed engineering design nor physical prototyping. This can be explained by the results from this paper which indicate that phase 1 (idea generation) would get the most submissions, phases 2 (concept development) and 5 (design evaluation) would get less submissions than phase 1 and phases 3 (engineering design) and 4 (physical prototyping) would get the lowest quantity of submissions. 

 To answer the question of how crowd participation depends on phase characteristics quantitatively and to make the picture of crowd participation clearer, we conducted an experimental study on crowd participation pattern over phases of a crowdsourced NPD project in a quantitative manner. Our findings indicate that the average submission rate for phase 1 (idea generation, S/V = 0.91%) is about 10 times higher than that for phase 4 (physical prototyping, S/V = 0.09%).

4.4 Guidelines for Application

According to the above discussions, the paper has suggestions for companies who want to execute crowdsourced NPD processes as follows 

Among phases, according to experimental data, phase 1 which is related to idea generation is the one that attracts the most solutions in terms of quantity. For a certain crowd, implementing phase 1 as a crowdsourcing project is more promising than other phases in terms of quantity of submitted solutions. Phases 2 and 5 can also gain a relatively high quantity of submitted solutions compared with phases 3 and 4 whose quantities are very low.

 Although phases 3 and 4 have a low gain of submitted solutions, they can still be utilized because companies can expect high quality submissions from well filtered solution providers. For a company who wants to apply crowdsourcing for phases 3 or 4, the company should divide the problem into mini-tasks so that less efforts and skills from the crowd are required. This will help to encourage the participation of the crowd to the project. The company can also combine the implementation of crowdsourcing with the use of its own product development team to gain better participation and performance.


By conducting an experiment on a real crowd of engineering related individuals, this paper investigated how the crowd responds to open calls for various product design and development phases. It is found that the level of complexity of a certain NPD phase (i.e., level of efforts, skills, knowledge it requires from the crowd) is an important factor that affects the number of individuals who participate in the phase.

 By conducting an experiment on a real crowd of engineering related individuals, this paper investigated how the crowd responds to open calls for various product design and development phases. It is found that the level of complexity of a certain NPD phase (i.e., level of efforts, skills, knowledge it requires from the crowd) is an important factor that affects the number of individuals who participate in the phase.

This paper provides practical guidelines for companies who want to conduct crowdsourcing for their NPD projects. When companies want to get a high rate of crowd participation in their NPD projects, they should apply crowdsourcing to phases 1, 2 or 5. If they want to apply crowdsourcing to phases 3 and 4, they should devide the problems into mini-tasks so that more participation can be obtained. By understanding the crowd participation pattern, companies can choose the suitable NPD phases to apply crowdsourcing. In other words, they can build a better environment for individuals to participate in crowdsourcing projects and benefit from their participation. 

There are also limitations to the results of this paper. First, in this paper, we consider only the crowd participation (i.e., quantity wise). The high participation rate might lead to a high chance of bringing innovation. Companies who want to grasp high quality submissions with crowdsourcing should consider seriously about developing filter mechanisms to filter qualified users as well as submitted contents. By this way, companies can benefit from implementing crowdsourcing for new pro-duct development. Second, in this paper, we assume that there is no correlation among crowd, product, process and reward factors. For the general case of possibly having correlations among some of these factors, another experiment is needed to investigate the general pattern. 

 Future work for this topic might include: 1) the relationship between crowd participation and other factors simultaneously, 2) the mathematical model that allows companies to predict crowd participation for a certain combination of {crowd, reward, product, process}, and 3) the quality of submissions from the crowd and influencing factors.


This work was supported by the project “Development of the Framework for Integrated Product Development to Implement Crowdsourcing under Mass Customization Environment” (Project No.: S-2011-A0004-00018) funded by Korean Ministry of Education Science and Technology. 


1.Albors, J., Ramos, J. C., and Hervas, J. L. (2008), New learning network paradigms: Communities of ob-jectives, crowdsourcing, wikis and open source, In-ternational Journal of Information Management, 28 (3), 194-202.
2.Bartl, M., Jawecki, G., and Wiegandt, P. (2010), Co-crea-tion in new product development: conceptual fra-mework and application in the automotive in-dustry, Proceedings of the R&D Management Con-ference: Information, Imagination and Intelligence in R&D Management, Manchester, England.
3.Bertoni, M. and Chirumalla, K. (2011), Leveraging web 2.0 in new product development: lessons learned from a cross-company study, Journal of Universal Computer Science, 17(4), 548-564.
4.Brabham, D. C. (2008), Crowdsourcing as a model for problem solving: an introduction and cases, Con-vergence, 14(1), 75-90.
5.Borst, W. A. M. (2010), Understanding crowdsourcing: effects of motivation and rewards on participation and performance in voluntary online activities, PhD Thesis, Erasmus Research Institute of Man-agement, Rotterdam, Netherlands.
6.Brondoni, S. M. (2010), Intangibles, global networks and corporate social responsibility, Symphonya Emerging Issues in Management, 2, 6-24.
7.Chesbrough, H. (2006), Open Innovation: The New Im-perative for Creating and Profiting from Technology, Harvard Business School, Boston, MA.
8.Chidamber, S. R. and Kon, H. B. (1993), A research retrospective of innovation inception and success: the technology-push, demand-pull question, Inter-national Journal of Technology Management, 9(1), 94-112.
9.Cooper, R. G. and Edgett, S. J. (2008), Ideation for product innovation: what are the best methods?, Visions, 32(1), 12-17.
10.Dekkers, R. (2011), Perspectives on Living Labs as in-novation networks, International Journal of Net-working and Virtual Organizations, 9(1), 58-85.
11.Dunphy, S. M., Herbig, P. R., and Howes, M. E. (1996), The innovation funnel, Technological Forecasting and Social Change, 53(3), 279-292.
12.Ebner, W., Leimeister, M., Bretschneider, U., and Krcmar, H. (2008), Leveraging the wisdom of crowds: designing an IT-supported ideas competition for an ERP software company, Proceedings of the 41st An-nual Hawaii International Conference on System Sciences (HICSS), Waikoloa, HI, 417.
13.Ebner, W., Leimeister, J. M., and Krcmar, H. (2009), Community engineering for innovations: the ideas competition as a method to nurture a virtual com-munity for innovations, R&D Management, 39(4), 342-356.
14.Elmquist, M., Fredberg, T., and Ollila, S. (2009), Explo-ring the field of open innovation, European Jour-nal of Innovation Management, 12(3), 326-345.
15.Fahling, J., Blohm, I., Krcmar, H., Leimeister, J. M., and Fischer, J. (2011), Accelerating customer integra-tion into innovation processes using Pico-Jobs, In-ternational Journal of Technology Marketing, 6(2), 130-147.
16.Greengard, S. (2011), Following the crowd, Communi-cations of the ACM, 54(2), 20-22.
17.Herstatt, C. and Lettl, C. (2000), Management of "tech-nology push" development projects, Working paper no. 5, Institute for Technology and Innovation Ma-nagement, Hamburg University of Technology, Ham-burg, Germany,
18.Hinchcliffe, D. (2007), Product development 2.0, Com-munications and Strategies, 65, 105-111.
19.Hinchcliffe, D. (2009), Crowdsourcing: 5 reasons it's not just for startups any more, ebizQ
20.Horton, J. J. and Chilton, L. B. (2010), The labor eco-nomics of paid crowdsourcing, Proceedings of the 11th ACM Conference on Electronic Commerce, Harvard University, MA, 209-218.
21.Howe, J. (2006), The rise of crowdsourcing, Wired Ma-gazine, 14(6).
22.Howe, J. (2009), Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business, Three Rivers Press, New York, NY.
23.Kim, H. J., Ahn, J. H., Tran, T., Nguyen, T., and Park J. Y. (2010), Analysis of ability of applying crowdsourcing for new product development pro-cess through case studies, Proceedings of 2010 KIIE Conference, Seoul, Korea.
24.Kleemann, F., Voß, G. G., and Rieder, K. (2008), Un (der)paid innovators: the commercial utiliza-tion of consumer work through crowdsourcing, Science, Technology and Innovation Studies, 4(1), 5-26.
25.Leimeister, J. M., Huber, M., Bretschneider, U., and Krc-mar, H. (2009), Leveraging crowdsourcing: ac-tivation-supporting components for IT-based ideas competition, Journal of Management Information Systems, 26(1), 197-224.
26.Malone, T. W., Laubacher, R. and Dellarocas, C. (2009), Harnessing crowds: mapping the genome of collec-tive Intelligence, Working paper no. 2009-001, MIT Center for Collective Intelligence, Cambridge, MA.
27.Mason, W. and Watts, D. J. (2009), Financial incentives and the 'performance of crowds', ACM SIGKDD Explorations Newsletter, 11(2), 100-108.
28.Panchal, J. H. and Fathianathan, M. (2008), Product realization in the age of mass collaboration, Pro-ceedings of ASME International Design Engineer-ing Technical Conferences and Computers and In-formation in Engineering Conference (IDETC/CIE 2008), Brooklyn, NY, 219-229.
29.Piller, F. (2010), Interactive value creation with users and customers, In Leading Open Innovation, Peter Pribilla Foundation, Munich, Germany,
30.Poetz, M. K. and Schreier, M. (2012), The value of cro-wdsourcing: can users really compete with profes-sionals in generating new product ideas?, Journal of Product Innovation Management, 29(2), 245-256.
31.Redlich, T., Wulfsberg, J. P., and Bruhns, F. L. (2008), Virtual factory for customized open production.
32.Schenk, E. and Guittard, C. (2009), Crowdsourcing: what can be outsourced to the crowd, and why?, Technical report, doc = halshs-00439256&version = 1.
33.Snow, C. C., Fjeldstad, O. D., Lettl, C., and Miles, R. E. (2011), Organizing continuous product develop-ment and commercialization: the collaborative com-munity of firms model, Journal of Product Innovation Management, 28(1), 3-16.
34.Sun, Y., Fang, Y., and Lim, K. H. (2012), Understanding sustained participation in transactional virtual communities, Decision Support Systems, 53(1), 12-22.
35.Ulrich, K. T. and Eppinger, S. D. (2008), Product Design and Development (4th ed.), McGraw-Hill Hi-gher Education, Boston, MA.
36.Ward, A., Liker, J. K., Cristiano, J. J., and Sobek, D. K. (1995), The second Toyota paradox: how delaying decisions can make better cars faster, Sloan Man-agement Review, 36(3), 43-61.
37.West, J. and Gallagher, S. (2006), Challenges of open innovation: the paradox of firm investment in open-source software, R&D Management, 36(3), 319-331.
38.Whitla, P. (2009), Crowdsourcing and its application in marketing activities, Contemporary Management Research, 5(1), 15-28.
39.Wightman, D. (2010), Crowdsourcing human-based computation, Proceedings of the 6th Nordic Con-ference on Human-Computer Interaction: Extend-ing Boundaries, Reykjavik, Iceland, 551-560. Taha, Z., Alli, H., and Abdul Rashid, S. H. (2011), Users involvement in new product development process: a designers' perspectives, Industrial Engineering and Management Systems, 10(3).
40.Taha, Z., Alli, H., and Abdul Rashid, S. H. (2011), Users involvement in new product development process: a designers' perspectives, Industrial Engineering and Management Systems, 10(3).