Evaluating

Criterion D: Evaluating

Example material from the MYP

Objective D: Evaluating

(i) Design detailed and relevant testing methods that generate data to measure the success of the solution.

By the end of year 5 you should be able to:

• design a wide range of effective tests to evaluate the solution against the requirements of the design specification (including expert appraisal, user trials, field testing and user observation)

Testing methodsAn effective and authentic measure of a design solution means that you tested against every aspect of the design specification. These tests can be classified as follows:

User observation

• The user is presented with the solution and is set a task to achieve with little or no guidance. The user’s interaction with the solution is observed and recorded.

User trials

• The user is presented with the solution and guided on how to use it. The user is asked questions as he or she interacts with the solution or is given a survey to complete. User trials may include focus groups.
• The design of interview or survey questions needs to be targeted to draw out responses that assess the solution against the specification.

Expert appraisal

• A person considered an expert in the use of similar products is presented with the solution, given time to interact with the solution and then interviewed on aspects of its success. The expert has particular knowledge and skills that allow him or her to make judgments on the success of the solution. The expert may be the client.

Field trial

• A field trial is a test of the performance of a solution under the conditions and situation in which it will be used. For example, an interactive information point (developed in HTML) for a museum exhibit may be tested by the exhibit visitors in the museum, structured as a user trial or user observation.

Performance testing

• The performance of a solution is tested under the conditions in which it would normally be used. Quantitative data is collected through a variety of tests such as:

1. destructive tests assessing impact strength or flammability
2. cyclic tests
3. measurement of physical properties such as weight and size
4. timed tests for web pages to load
5. ease of navigation through an interactive story, game or website.

                    --------------------------------------------------------------------------------------------------------

Collecting dataBoth types of data collected through testing are considered primary sources of data.
Qualitative data deals with quality and is data gathered as descriptions. This data tends to be subjective and can be converted to a numerical value, eg “I like the overall shape of the solution, it fits my hand well, I would give it a 9 out of 10 for comfort” or “The layout of the webpage looks intuitive, it looks easy to navigate and the use of negative space makes it clear. I would rate the clarity of the page as 8 out of 10.”
Tests that can be used to obtain qualitative data include:

• using a questionnaire to find out if the target audience likes the look of a product
• surveying students to find out which parts of a video game they found too easy and which were too difficult
• interviewing an expert after he or she has interacted with a solution
• performing a user trial by giving a toy to children to play with and observing reactions.

Quantitative data deals with quantity and is gathered as definite values, typically a numerical value. This data is objective and can be measured, eg “All information in the database has to be a maximum of 3 clicks away” or “The overall cost of the materials can be no more than €45.”
Tests that can be used to obtain quantitative data include:

• timing users who are tasked with finding a particular piece of information on a website
• measuring a product to ensure it is the correct size and within weight limits
• beta-testing interactive media to find bugs
• running performance tests to determine the strength of a product
• checking the capacity of a storage device
• counting the number of hits on a website over a set period of time.

Test - comparison with existing product

My product

From the surveys, I can conclude that my product met most of my design specification. My product has three sections for putting different writing utensils and all sections are at least 11cm deep. Every section can stored more than 20 pencils and I have a drawer that can put more than 3 erasers. My product is made out of plywood and is joined well using glue and nails. I finished my product in time (10 create classes). For my fourth design specification (The design theme must be aesthetically pleasing to a female teenager around 14 to 16 years old, the colour theme will be plain solid colour), 6 people said it looks excellent and 5 said it looks good. Some like its simplicity and they said that the colour theme and popsicle sticks make the product looks pretty. However some also say that it is a little bit too plain and not very attractive. What most students like about my product is its function and how it can stored many stationary. They also like the design: circular form and decorations (popsicle stick). What they said can be improved id the drawer because it looks a bit small and it didn’t fit well, therefore it is hard to pull out. They also said I should make each sections less deep and wider because it is quite hard to pull the pencils out. They also said to change the colour theme from white to yellow or add more colour in the front (because it is too simple).

                                                                             Feedback from users example

                       

---

(ii) Critically evaluate the success of the solution against the design specification

By the end of year 5 you should be able to:

• carry out a wide range of effective tests to generate qualitative and quantitative data
• explain the accuracy of the data gathered throughout testing
• present the analysis of the data in a clear and concise way
• evaluate the success of the solution against the requirements of the design specification
• explain weakenesses and limitations of the solution based on the analysis of data collected throughout objective and subjective testing against the requirements of the design specification

When critically evaluating the success of a solution, you must objectively judge the solution against the specifications. This objective evaluation comes from analysing the data gathered by carrying out the tests designed.
You must analyse this data and make conclusions that identify strengths and weaknesses of the solution, along with opportunities for further development.

Specification	Testing method	Evidence from tests
-----------------------	-----------------------	----------------------- -----------------------
-----------------------	-----------------------	----------------------- -----------------------

                                              Testing against the design specifications template
Test against design specifications

Specifications	Yes/No	How is it met?
Must have at least 3 sections for putting different types of writing utensils	5/5	There are three sections for putting different types of writing utensils: one big section and two small sections
Must have enough space for storing at least 20 pencils	5/5	One of the small sections of my product can store 40 pencils
Must have 1 drawer for storing erasers that can store at least 3 small erasers	5/5	My drawer can stored 4 erasers
The design theme must be aesthetically pleasing to a female teenager around 14 to 16 years old. The color theme will be plain solid color	3/5	According to the survey results, 6 people said the design is excellent and 5 said it is good. Some say that the product is too plain
The depth of each section must be at least 11 cm drop	5/5	The long section is 11cm deep and the small ones are 14cm deep
Must be made out of wood	5/5	The product is made of plywood
Must be joined well using glues and screws/nails	5/5	The product is joined using latex glue and nails (from nail gun)
Must be completed within the time given (the create part)	5/5	The product is completed within the given time below

                                      Recording the results of tests against the design specifications example

---

(iii) Explain how the solution could be improved

By the end of year 5 students should be able to:

• explain, in detail, how the identified weakenesses and limitations of the solution could be improved
• create annotated sketches and adapt the design specification to show how the solution could be improved

How can students identify how the solution could be improved?Through the objective evaluation of your solution, you should have identified weaknesses in your solutions. These weaknesses provide opportunities for further development and allow you to consider how you would refine your solution further.
Students can suggest these improvements in many forms, including:

• written text—paragraphs or tables
• diagrams and charts
• annotated photographs/screenshots of the prototype
• sketches

Amended drawings/diagrams/storyboards etc. Adding the name of the places in the pictures (Applies to both model and final)	This was suggested by the ministry of tourism and it is a valid improvement. This would be beneficial to my product because it would provide tourists with some information on what these places are called and it would help them later for further inquiries into Phnom Penh. It is also a good idea in general because it also advertises certain areas of Phnom Penh.
Adding the Cambodian tourism website (Applies to both the model and full size display)	This was also suggested by the ministry and is a good idea as well. This will provide tourist with easy contact information. Most airports have Wi-Fi in them, thus, they are able to check out the website easily and find out more if they wish to. This is also smart.
Add some facts or other information about Phnom Penh such as transport (Applies to both the model and full size display)	This would allow the tourists to see more about Phnom Penh without having to do further inquiries. This is helpful for them because they can decide just by seeing the booth. It may also provide for something more to look at in the booth because my booth is rather simple.
Size of the lotus needs to be bigger (Applies to model)	The lotus on the model was a little too small that it would not look very good on the full size display. Therefore, it would be better to have a bigger lotus because it would look more impressive and would be a better central attraction. The lotus was small because of the sizes of paper used to create it. These sizes could be made bigger.

                                                        Presenting possible improvements to a solution examples

Strengths	Weakeness	Improvements
Tastes of the dishes The taste of both the dishes were excellent according to all of my target audiences. The pasta had a creamy flavour with juicy mushrooms, mock chicken which adds to the texture and soft spinach which balances out the dish. The salad had a wide range of vegetables which when eaten provide a crunchy experience for our tasters, but when combined with the sour of the lemon and thickness of the feta cheese, it tastes very sublime. Nutrients in the dishesThe nutrients in the dish ranged from protein, carbohydrates, vitamins, minerals to dairy products. Even though the dish was for a vegetarian, it provided a sufficient amount of nutrition which is what an athlete needs in order to perform. Organization and Time-managementWe organized the cooking fairly and equally so that we could be efficient when cooking the dishes as well as have time to clean up as we go. This really benefitted us at the end because we finished just on time, even though we encountered a few problems during the end. When we faced any problems, we immediately fixed them and got on with the cooking to save time and also finish on time.	Appearance and texture of the pasta dish The appearance and texture of the pasta was not that great as we would have liked it to be. The design clearly wanted our pasta to be surrounding the sauce, but a minor mistake made us change the appearance so I am disappointed. The texture, consequently, was also ruined due to our poor quality sauce which was too thick. We were hoping to fix that mistake from the first time, but unfortunately, it did not work out the second time either. Cooking errors We did not identify some things correctly such as when the water was actually boiling because we accidentally put the pasta in non-boiling water so we need to be more focussed to fix that.	If I had a chance to redo this dish, I would definitely have done this twice and used the first time as a learning experience to fix all my mistakes and not repeat them again such as the thickness of the pasta cream sauce. I will also try and practice basic skills such as boiling at home beforehand so I can perform it correctly in school. I think I just need to cook more to get more experience with these sort of things which will lead to a perfect dish. Since pasta is eaten quite often, maybe next I would try and create my own dish that will excite my target audience a bit and give them something different to what they usually have.

                                                                    Presenting possible improvement to a solution example                                                    Figure 72 - Suggested improvements example: Clock

---

(iv) Explain the impact of the solution on the client/target audience

By the end of year 5 students should be able to:

• explain the impact of the solution on the client or target audience

To identify or predict the impact that a solution will have on a client or target audience, you must refer to the original problem, the design brief, the specification and the evaluation. You should use these aspects to draw conclusions about how well the design brief has been met. These conclusions may be presented in written form, as a list or as a table.The following questions may help you to explain the impact of the solution on the client or target audience.

• To what extent has the client’s or target audience’s problem been solved?
• How does this solution improve the client’s or target audience’s situation?
• To what extent has the design brief been met?
• Are there any negative effects this solution could have?

Criterion C: Creating the Solution

Objective C: Creating the solution

(i) Construct a logical plan that describes the efficient use of time and resources, sufficient for peers to be able to follow to create the solution.

By the end of year 5 you should be able to:

• construct a series of logical steps to create the solution, using charts, diagrams and text that include aspects of quality control and quality assurance
• construct a plan to create the solution that makes effective use of resources and time
• construct a clear and concise plan that peers will be able to follow to create the solution

Objective C strand 1

In objective C, your role switches from designer to prototype-developer or manufacturer. Objective C focuses on the realization or creation of a solution to a problem.
One of the most difficult aspects of design is to take an idea from paper and create a product that someone can interact with: a solution to a problem. In objective B, you developed planning drawings and/or diagrams to clearly depict what you are making. In this first strand of objective C, you detail how they will make your solution.
By constructing a logical plan that details the steps required to make a solution, you will demonstrate your knowledge and understanding about how solutions can be made.

What are resources?

Resources can be categorized in two ways: materials and tools.

 When identifying digital materials, you should consider text, fonts, images, audio, video, animation, sprites and icons.

For digital design, tools include both software and hardware (input, processing and output devices). For product design, tools include: hand tools, machinery, CAD (Computer Aided Design) software, EDA Electronic Design 

Quality control and assurance

Quality assurance covers all materials from design to documentation. It includes the regulation of quality of raw materials, assemblies, products and components, services related to production and management, and inspection processes.

Quality control manifests itself in the development checking and testing of systems to ensure that products or services are designed and made to meet or exceed specifications.

Examples of plans

Step-by-step plan

Step	Process (including quality control, health and safety considerations)	Tools	Materials	Time to complete
1
2

                                                                  Step-by-step plan template

Step	Description of the tasks	Duration	Resources; Tools,  & Materials required	Place/Venue
1	Create background.  I will design my background to show the color scheme of the client's business of blue and yellow colors.	1 hour	Animation software: animaker.com	In class
2	Create props to show the different furniture; counter, fridge, cashier desk and shelves	50 min	animaker.com	In class
3
4
5
6

                                                                                                  Step-by-step plan example
Gantt chart/slippage chart

A Gantt chart is a project-management tool used to provide an overview of a process. A Gantt chart divides the manufacture of a product into small tasks. It indicates the time estimated for each of these tasks and the resources required. It does not provide a detailed set of instructions, but instead is an overview to self-assess progress with a project. You should indicate when each task is complete.

Gantt chart example

• Gantt charts using Excel and free online software

A slippage chart goes one step further than the Gantt chart in that it has room below each task to mark when it was completed (in green), allowing you to identify when they are behind in their plans and take action to get back on schedule.

Slippage chart example

The following questions can help you consider your planning.

• Does your plan contain a sequence of logical steps?
• Is the use of resources explained clearly?
• Have you made sure you have enough time to complete your product?
• Have you given time for practising or learning a new skill?
• Have you allowed extra time in case something goes wrong?
• Have you considered alternative ways of creating the solution?
• Have you planned for testing at appropriate times in the manufacturing process?

---

(ii) Demonstrate excellent technical skill when making the solution

At the end of year 5 you should be able to:

• develop manipulative skills to effectively use a wide range of tools, techniques and equipment
• use a wide range of tools and equipment proficiently to create high-quality solutions
• ensure a safe working environment for themselves and others

To determine the correct level of technical skill which you are operating at, teachers need to consider the complexity of skill demonstrated as well as the level of guidance that the teacher needs to provide in order for you to complete the task.
When determining the complexity of skill demonstrated by you, a number of indicators will be considered.

• Accuracy of the solution: Have all parts been made as detailed in the plan? Have all parts been made accurately?
• Final aesthetic quality of the solution: Has the student given appropriate attention to detail, considering the final overall look and feel of the solution?
• Assembly of the solution: Do all components fit together or combine as detailed in the plan?

When determining the level of guidance and support needed, teachers will consider that you:

• can be given initial guidance and demonstrations by the teacher
• can practise the processes as required
• must work independently when making the solution, if they are to achieve the higher levels
• should demonstrate safe working practices.

For digital design, the level of technical skill demonstrated can be shown through the functionality of the assembled solution. For product design, the level of technical skill demonstrated can usually be shown through the functionality and the final finish of the assembled solution.

---

(iii) Follow the plan to create the solution, which functions as intended

By the end of year 5 you should be able to:

• independently follow the plan to produce the solution, which functions as intended
• manage time effectively to produce the solution to the deadline stated in the plan

You will demonstrate that the planning process was sufficiently thorough if the final product matches the planning drawing and no changes were made to the plan, in whatever form it was presented. If the drawings/diagrams and the plan match what has been made, then you have followed the plan.
If a you do not follow the plan, it will usually result in the solution not fully meeting the specification and/or matching the design detailed in the drawings/diagrams.

---

(iv) Fully justify changes made to the chosen design and plan when making the solution

By the end of year 5 you should be able to:

• justify, through the use of drawings/diagrams, any changes made to the design while making the solution and how they affected the plan

As you implement your plans, you will often come to a point where you will have to modify the plan or the design. You will need to ensure that you detail and justify any changes that you make during the creation process.

If you do not make any changes to your product or plan, this strand does not have to be addressed with evidence. However, if any changes are made, 

you must address this strand.

There are two main changes and modifications I have made to my original idea: The hole on the edge of the small wooden pieces (used for the numbers) used to help lift the pieces was originally a square of dimensions 15mm x15mm. Nevertheless, I have found a better solution that is easier to create and is more practical. I will instead create a circle of radius 10mm. This will be done using a bench drill which is easier and faster than cutting it in a square shape using a saw, especially since I will be creating 20 pieces of these. In the original plan, the snake was supposed to be painted on the top 600 x 420mm wooden piece, with the holes for the number pieces being in it. Nevertheless, I have found a better solution which is to cut out the snake (with the dimensions listed in the ‘Parts list’ page) and create the holes in the snake itself. This would not only make the product more attractive, it would additionally make its creation more practical and easier since it would be easier to rotate the snake than the whole board when creating the holes.

Changes made to design example: Educational toy

On the home page… The link has moved from the very bottom to almost near the top, I had done this for aesthetical reasons. The quote, instead of having it centered, it is now flushed right, this was because I needed to create more of a visual balance. Other than that, it appears the same in the design as it does in the final product.  Design  Solution                                                                   Changes made to design example: Website

Changes made to design example: Making a box

Modification to my Design Now that I have made a plan to follow and thought about all the materials, tools I am going to use and how much time the project is going to take, some features of my design changed. To start with, I changed the thickness of the circle of the sun which is where I am going to locate the clock mechanism. This is because the part of the clock mechanism that has to be drilled through the material is 6cm long. So, it is going to take a long time to drill a whole about 14cm deep on an 18cm thick material. I thought it would take less time drilling a whole about 2 cm deep on an 8cm thick material Now that I thought about how long it is going to take to create stickers, I decided make more Number Stickers. Before I made the plan, I thought it would take the longest time to create the stickers on the vinyl cutter; however, I realized it actually doesn’t. I was going to only make four stickers: 12, 3, 6 and 9. But, since it doesn’t take so much time creating them, I decided to make all twelve stickers. I also realized that while I am waiting for the stickers to be cut out by the vinyl cutter, I could be using the time and start working on the next step which is putting the clock mechanism on to the clock design. The last change I have made is to use spray paint for one or two parts. If I use paint, I have to paint several coats and dry them. Same goes for spray paint; however, if you compare the time on how long it takes to paint and spray a material once, to spray paint takes less time.                                                                                          Modifications to a design example

---

(iv)(a) Present the solution as a whole

At the end of year 5 you should be able to:

• present the solution in an appropriate format that shows all details of construction

A digital design product will be presented as either an executable file or readable file.

A product design outcome will be presented as a set of photographic images, which must show detail and scale.

As we journey into our third leg of the design cycle, your task will entail the following;

i) Construct a detailed and logical plan, which describes the efficient use of time and resources, 

sufficient for peers to be able to follow to create the solution. (1-3 pages)

ii) Create the product that demonstrates excellent technical skills usage. (ii and iii, 6 pages)

In this stage, there should be clear evidence that the plan is followed, the final product 

should meets the design specifications and should be a close as possible to your chosen 

design, and that the solution functions as intended and is presented appropriately.

iii) follow the plan to create the solution, which functions as intended

iv) fully justify changes made to the chosen design and plan when making the solution ( 2 pages or 4 minutes of video)

a. present the solution as a whole.

It's expected that a dated process journal is kept with detailed screenshots justifying any 

improvements in your product development progress. The screenshots will normally show 

how a particular complex techniques are applied before and after.

Criterion C: Creating the solution
0	The student does not reach a standard described by any of the descriptors below.
1 — 2	The student demonstrates minimal technical skills when making the solution, creates the solution, which functions poorly and is presented in an incomplete form.
3 — 4	The student constructs a plan that contains some production details, resulting in peers having difficulty following the plan, demonstrates satisfactory technical skills when making the solution, creates the solution, which partially functions and is adequately presented, outlines changes made to the chosen design and plan when making the solution.
5 — 6	The student constructs a logical plan, which considers time and resources, sufficient for peers to be able to follow to create the solution, demonstrates competent technical skills when making the solution, creates the solution, which functions as intended and is presented appropriately, describes changes made to the chosen design and plan when making the solution.
7 — 8	The student constructs a detailed and logical plan, which describes the efficient use of time and resources, sufficient for peers to be able to follow to create the solution, demonstrates excellent technical skills when making the solution., follows the plan to create the solution, which functions as intended and is presented appropriately, fully justifies changes made to the chosen design and plan when making the solution.