Example Review Forms: Formal K-12 Education, Educational Resource, Postsecondary Resource, Science Content, Informal Education
Review Criteria Used in These Forms
- Materials are appropriate, complete, and effectively presented and of a high production/design quality.
- The product is easy to use and free from technical difficulties.
- Scientific accuracy - the content was developed with subject matter experts and is presented accurately.
- Instruction and Assessment - Materials emphasize effective instructional practices and pedagogical approaches and provide appropriate student assessment.
- National Education Standards (formal education products) - Instructional material for K-12 formal education support National Education Standards.
- Material is relevant to NASA-unique Science Mission Directorate (SMD) Content.
- Safety- The materials address NASA's first priority: Safety. Risk(s) from proposed materials/equipment/protocol etc. , and risk mitigation procedures are identified. For websites targeted to children under 13, the site requires parental permission before collecting personal information.
- Overall Assessment
- Final Recommendation
- In applying the criteria, several guidelines should be considered by reviewers as well as developers:
- It is important to note that with all of these criteria, the labeled bullet points are given as ranges of examples.
- Not all examples included in the criteria descriptions are expected to be addressed by every product. Therefore, when reviewing the materials, one should not use the suggested examples as a checklist.
- Some products being reviewed may still be in an unfinished state so that recommendations from the education product review may be implemented along with final design.
- For exhibits and other products where the reviewer cannot access the actual item, documentation describing the product, its implementation, and status should be provided to the reviewer.
- Although not criteria, materials should address 508 Compliance and consider Universal Design (UD).
- Compliance to Section 508 guidelines (https://www.section508.gov/) is required. This aspect can be completed after the other dimensions of the review are addressed.
- Universal design (https://www.w3.org/WAI/intro/wcag.php) is an aspirational goal for products submitted to this review. Comments on aspects of the product that address these ideals are welcome but not required.
For more information on universal design, see the seven general principles of Universal
Design found at: https://projects.ncsu.edu/ncsu/design/cud/about_ud/udprinciplestext.htm
Universal Design for Learning (UDL) has three main principles that are used in curriculum design:
- Multiple means of representation—presenting information in different ways;
- Multiple means of expression—allowing students to express knowledge in different ways; and
- Multiple means of engagement—stimulate learning and engagement in different ways.
For expansion on the elements of UDL see:
- NSTA Science for Students with Disabilities http://www.nsta.org/disabilities/design.aspx
- National Center on Universal Design for Learning http://www.udlcenter.org/aboutudl/udlguidelines_theorypractice
- Additional useful reference information regarding science literacy can be found in documents such as:
- Essential Principles of Climate literacy (https://www.climate.gov/teaching/essential-principles-climate-literacy/essential-principles-climate-literacy )
- Ocean Literacy Principles (https://oceanservice.noaa.gov/education/literacy.html )
- Atmospheric Science Literacy Framework (https://scied.ucar.edu/atmospheric-science-literacy-framework )
- Earth Science Literacy Principles (http://www.earthscienceliteracy.org/ )
Criterion 1: OVERALL CONTENT - The materials are appropriate, complete, effectively presented, and of a high production/design quality.
1.A. Materials are appropriate for the age, grade, and maturity of the target audience.
1.B. Current, up-to-date information is provided.
1.C. The material is free from production errors (e.g., misspellings, typos, grammatical, and editorial errors).
1.D. Instructional or explanatory materials are effective and well written; acronyms and terms are clearly explained; information is presented in a logical and organized manner; answers are provided to all specific questions asked.
1.E. Graphs, charts, images, and animations are clearly labeled/indicated/narrated and include color keys where appropriate, describe units of measure (e.g., what these units mean and how they compare to familiar examples), and provide clear distinctions between scientific data and simulated data/artist renditions.
1.F. Visuals/images are crisp, clear, and/or high-resolution. Video/audio quality is high.
1.G. Where used, learning technologies are effectively employed and integratedCriterion 2: EASE OF USE - The product is easy to use and free from technical difficulties.
2.A. The user interface is easy to understand.
2.B. Instructions are easy to follow, clear, and complete.
2.C. The product is quick loading, user friendly, well organized, and structured.
2.D. The product is easy to navigate, meaning that there are not too many levels to click through; it is easy to move forward, backward, and "escape" (e.g., back to the home page, to quit, etc.); and updates are easy to find.
2.E. Documentation and any technical requirements for using the resource are specified.
2.F. The product is free from technical difficulties (e.g., doesn't freeze, no error messages, links to WWW sites are up-to-date). For example, on websites image files load quickly, text is legible, and the background does not interfere with reading.
2.G. Where appropriate, useful online help is provided.
2.H. Relevant tools are provided for using the product (e.g., suggestions for use in the classroom, links to needed software, etc.).
Criterion 3: SCIENTIFIC ACCRUACY- The content was developed with subject matter experts and is presented accurately.
3.A. The material is free from content errors (e.g., scientific and mathematical inaccuracies, incorrect facts or statements, theory and fact are adequately distinguished).
3.B. Text, figures and images do not create or reinforce misconceptions.
3.C. The metric system of weights and measures is consistently used (e.g., Celsius, grams, liters, meters) or metric equivalents are provided. For example: 10 m (32.8 ft) or 28℃ (82.4°F).
3.D. Specific and relevant references for further investigation/information are provided.
Criterion 4: INSTRUCTION AND ASSESMENT- The materials emphasize effective instructional practices and pedagogical approaches and provide appropriate student assessment.
4.A. Observable events, problem-centered tasks, and concrete experiences introduce key concepts of science.
4.B. Learners are encouraged to explore ideas and events, to construct relationships and patterns, and to make and test hypotheses.
4.C. Learning activities offer opportunities for students to revise or dispel prior conceptions, to build on existing knowledge and to create or construct new knowledge.
4.D. Opportunities are provided for learners to apply concepts. Materials provide opportunities for interactive learner participation (e.g., hands-on experiences, small group activities, communicating and sharing information is promoted).
4.E. Product is designed to foster outcome-based learning goals.
4.F. Materials include assessment suggestions and scoring criteria that are appropriate for the type of product and/or audience, including assessments that require students to apply concepts, analyze or synthesize information, or evaluate situations (i.e., to use higher order thinking skills). Assessment of prior knowledge is included as appropriate.
Criterion 5: NATIONAL EDUCATION STANDARDS- Instructional materials for K-12 formal education support National Education Standards..
5.A. The materials are well-aligned to education standards.
5.B. Content and practices align to the standards and specific student outcomes are identified.
5.C. In general, it is expected that NASA formal education materials will support national standards. In some cases, it may be appropriate to also identify relevant state standards. These standards could include:
· A Framework for K-12 Education
· AAAS Benchmarks for Science Literacy
http://www.project2061.org/publications/bsl/online/index.php
· Common Core State Standards for Mathematics
http://www.corestandards.org/Math/Practice/
· Common Core State Standards for English Language Arts
http://www.corestandards.org/ELA-Literacy/
· International Society for Technology in Education (ISTE) – technology education standards
https://www.iste.org/standards
· International Technology and Engineering Educators Association (ITEEA) Standards for Technological Literacy
https://www.iteea.org/39197.aspx
· Next Generation Science Standards (NGSS)—see the note below regarding the EQuIP Rubric.
https://www.nextgenscience.org/news/equip-rubric-science-released
For the EQuIP Rubric: Only if the product is a larger unit of study designed for NGSS the developer should complete the Fillable pdf or Word document of the EQuIP Rubric at http://www.nextgenscience.org/resources/equip-rubric-lessons-units-science. There are additional helpful resources located at this site. This form should be submitted by uploading the completed form to the product submission form along with uploaded or linked product documents.
Criterion 6: The material is relevant to NASA-unique Science Mission Directorate (SMD) content.
NASA SMD products/programs should be centered on and draw upon NASA’s unique assets in Earth or Space Sciences: content/information (acquired through NASA science and technology programs and missions); facilities and tools (including observational datasets); or people (including NASA employees and NASA-sponsored scientists, technical and engineering experts) in at least one of the following areas:
- Earth Science
- Heliophysics
- Planetary Science
- Astrophysics
EARTH SYSTEM SCIENCE
NASA’s strategic objective in Earth science is to advance knowledge of Earth as a system to meet the challenges of environmental change, and to improve life on our planet.
NASA’s Earth science program seeks to answer the following questions:
- How is the global Earth system changing?
- What causes these changes in the Earth system?
- How will Earth’s systems change in the future?
- How can Earth system science provide societal benefits?
These science questions translate into seven overarching science goals:
- Advance the understanding of changes in the Earth’s radiation balance, air quality, and the ozone layer that result from changes in atmospheric composition (Atmospheric Composition)
- Improve the capability to predict weather and extreme weather events (Weather)
- Detect and predict changes in Earth’s ecological and chemical cycles, including land cover, biodiversity, and the global carbon cycle (Carbon Cycle and Ecosystems)
- Enable better assessment and management of water quality and quantity to accurately predict how the global water cycle evolves in response to climate change (Water and Energy Cycle)
- Improve the ability to predict climate changes by better understanding the roles and interactions of the ocean, atmosphere, land and ice in the climate system (Climate Variability and Change)
- Characterize the dynamics of Earth’s surface and interior, improving the capability to assess and respond to natural hazards and extreme events (Earth Surface and Interior)
- Further the use of Earth System science research to inform decisions and provide benefits to society
HELIOPHYSICS
NASA’s strategic objective in heliophysics is to understand the Sun and its interactions with Earth and the solar system, including space weather.
The domain of heliophysics ranges from the interior of the Sun, to the upper atmosphere and near-space environment of Earth (above 50 kilometers), and outward to a region far beyond Pluto where the Sun’s influence wanes against the forces of interstellar space. Earth and the other planets of our solar system reside in this vast extended atmosphere of the Sun, called the heliosphere, which is made of electrified and magnetized matter entwined with penetrating radiation and energetic particles.
To increase our understanding of the heliosphere, NASA seeks to answer fundamental questions about this system’s behavior:
- What causes the Sun to vary?
- How do geospace, planetary space environments, and the heliosphere respond?
- What are the impacts to humanity?
To answer these questions, NASA’s Heliophysics Division is implementing a program to achieve three overarching science goals:
- Explore the physical processes in the space environment from the Sun to the Earth and throughout the solar system
- Advance our understanding of the connections that link the Sun, the Earth, planetary space environments, and the outer reaches of our solar system
- Develop the knowledge and capability to detect and predict extreme conditions in space to protect life and society and to safeguard human and robotic explorers beyond Earth.
PLANETARY SCIENCE
NASA’s strategic objective in planetary science is to ascertain the content, origin, and evolution of the solar system and the potential for life elsewhere.
We seek to answer fundamental questions:
- How did our solar system form and evolve?
- Is there life beyond Earth?
- What are the hazards to life on Earth?
These important questions have been translated into the following science goals:
- Explore and observe the objects in the solar system to understand how they formed and evolve.
- Advance the understanding of how the chemical and physical processes in our solar system operate, interact and evolve.
- Explore and find locations where life could have existed or could exist today.
- Improve our understanding of the origin and evolution of life on Earth to guide our search for life elsewhere.
- Identify and characterize objects in the solar system that pose threats to Earth, or offer resources for human exploration.
The Planetary Science Division includes programs with three major classes of mission destinations:
- Inner solar system – Earth’s Moon, Mars and its satellites, Venus, and Mercury
- Outer solar system – Jupiter and its rings and moons, especially Europa; Saturn and its rings and moons, especially Titan and Enceladus; Uranus and its moons; Neptune and its moon Triton; the dwarf planet Pluto and its small moons; and other Kuiper Belt Objects
- Small bodies in the solar system, Comets, asteroids, and the dwarf planet
- Ceres in the asteroid belt
ASTROPHYSICS
NASA’s strategic objective in astrophysics is to discover how the universe works, explore how it began and evolved, and search for life on planets around other stars.
Three broad scientific questions and goals emanate from this objective.
• How does the Universe work? Probe the origin and destiny of our universe, including the nature of black holes, dark energy, dark matter and gravity.
• How did we get here? Explore the origin and evolution of the galaxies, stars and planets that make up our universe.
• Are we alone? Discover and study planets around other stars, and explore whether they could harbor life.
Criterion 7: SAFETY-The materials address NASA’s first priority: Safety. Risk(s) from the proposed materials/equipment/protocol etc., and risk mitigation procedures are identified. For websites targeted to children under 13, the site requires parental permission before collecting personal information..
7.A. Websites for children under 13 that collect personal information from children or general audience Websites that collect personal information from children (information that would allow someone to identify or contact the child), must comply with the Children’s Online Privacy Protection Act (COPPA). For more information, see http://www.ftc.gov/ogc/coppa1.htm.
7.B. The product is inherently safe. Any potential risk areas are identified so they can be mitigated.
7.C. When considering the adequacy of safety risk identification, communication and mitigation, the reviewer should consider whether the current educational product:
o Confirms that this science educational product was reviewed to assure conformance with standards of applicable subject matter scientific excellence, integrity and objectivity, but that no review was done with respect to Consumer Product Safety Commission (CPSC) rules and regulations or any other safety science related law, rules or regulations. If a CPSC review or other safety review has been complted, note this in the review comments;
o Conforms with the guidance set forth in the National Science Teachers Association (NSTA) Policy Position on Safety and School Science Instruction http://www.nsta.org/about/positions/safety.aspx (it is not a requirement that a product included this statement, but it is a good practice);
o Identifies the professional practice standard for science teachers set forth in document entitled “NTSA – Duty or Standard of Care” http://static.nsta.org/pdfs/DutyOfCare.pdf, by the NSTA Safety Advisory Board, April 2014, and any updates thereto;
o Ensures that the protocol clearly notes that human subject testing criteria must be considered and comply with applicable law? This applies if student or other human derived data are an integral part of the activity (e.g. the use of humans test subjects, data derived from tasting, smelling, or ingesting any substance, or when any other physiological responses are evaluated).
o Makes it clear that ultimately it is the responsibility of the science teachers and school administrators to use appropriate legal standards and better professional practices under duty of care to make the science laboratory safer.
o Makes clear that it is the SCHOOL, SCHOOL SYSTEMS, LOCAL, STATE OR FEDERAL LAWS, REGULATIONS, CODES AND PROFESSIONAL STANDARDS that set the safety criteria and standards that apply to the science educational product design and use; and encourage the teacher to check with their local authorities to get approval for use of such science educational products in their classroom.
o Makes clear that the science education product is provided as-is without warranty with respect the safety rules, regulations and laws.
7.D. Are safety measures prescribed by the activity appropriate for the specified ages/grade levels? Can the relevant safety risk be communicated and managed with the target age group such that any mitigation steps can be enforced and not easily bypassed? If not, what ages/grade levels would be appropriate for this science education product?
7.E. What, if any, safety equipment is identified (safety goggles, gloves, adequate ventilation, earplugs, dust masks, footwear, helmets, etc.)? If properly used would the safety equipment reduce risk of injury or other damage and will the target age group likely be able to successfully use such safety equipment properly with the student to teacher ratio common to the target age group?
7.F. Are there any attire limitations included in the instructions, for example if loose clothing could be a hazard or closed shoes are required. If so, are instructions clear that students whose attire does not conform to the safety criteria must be excluded from participation in the science education activity?
7.G. If observing the daytime sky is involved, is instruction to never look directly at the sun included? If observing Sun-related events are involved, are safe solar viewing apparatus provided along with training and instruction in their proper use?
7.H. When foods are used (consumed or not), have appropriate steps been taken to address and mitigate risks for those with food allergies, and from contamination of food, intentionally or unintentionally? Is the need for hand washing and container sterilization and other relevant food safety criteria made clear?
7.I. Is instruction provided regarding injury mitigation when running, hiking, or other exercise, etc. is involved in the lesson?
Criterion 8: Overall Assessment
Assess the materials overall. Provide additional feedback on anything (good or bad) that was not addressed in the above criteria.
Criterion 9: Final Recommendation
Recommended: The product is an excellent candidate for broad distribution or availability, for example, at education conferences, on a Web site, etc.
Recommeneded as is, with revisions on next printing/edition: Note to reviewers: the material may not be re-printed/revised. Do not select this option if you believe the revisions are necessary before continued distribution/availability by NASA. (Provide notes on recommended revisions.)
Recommended--Distribution through Teacher Workshops: The product is recommended for distribution through developer conducted teacher training sessions, workshops, online workshops, or online training videos. Most teachers would require some introduction or training for effective use.
Recommended--Limited Audience: This product would be useful to limited audience. For example, very advanced users, lower/higher education level than originally identified, a limited topic area/course, a limited geographic area, etc. (Provide notes describing the audience.)
Needs Minor Revisions: The overall approach is very good, but minor revisions are needed before the product is distributed or recommended by NASA. "Minor" is not meant to indicate the revisions are not important, but that they can be made easily. The material is generally free of scientific errors/misconceptions or obvious pedagogical problems; if there are errors they can be easily corrected. These products are not required to go back through the panel review process once they have been revised.
Examples of minor revisions include:
- Few text edits/minor text additions (e.g., grammatical errors, misspellings and typos).
- Minor corrections to images or graphics.
- Minor re-organizing of the material.
- Online products: a few broken links on a Web site (it is "minor" due to ease of correcting).
- Physical products: broken Web links, which are not critical to implementation (e.g., links for further reading/information).
Needs Medium Revisions: The overall approach is sound, but revisions are required before this product is distributed or recommended by NASA. Depending on the extent of revisions to be made, NASA education program managers may require these products to go back through the panel review process after they have been revised.
Examples of medium revisions include:
- Large number of text edits/additions throughout the material (e.g., grammatical erros, misspellings and typos).
- Corrections to more than a few images or graphics.
- Sections that suffer from organizational or presentational difficulties.
- Online products: numerous broken links on a Web site.
- Physical products: several broken Web links, which are not critical to implementation (e.g., links for further reading/information).
- Few scientific errors/misconceptions (perhaps limited to a single section, chapter, unit or lesson).
- Isolated sections, chapters, units of lessons that contain obvious pedagogical problems.
Needs Major Revisions: The product has potential, but major revisions are required to the overall approach before it is distributed or recommended by NASA. Products that are recommended for "major revisions" are required to go back through the panel review process after they have been revised.
Examples of major revisions include:
- Numerous or major scientific errors/misconceptions.
- Serious problems with the overall approach to the subject, organization, structure or presentation that renders it ineffective or difficult to use.
- Pedagogical appraoch needs significant work for the intended audience (e.g., reading level, cognitive appraoch, scope/sequence, etc).
- Broken links in a printed document, which are critical to the implementation of the product.
Not Recommended: NASA should take this product out of circulation/not release it as a Science Mission Directorate science education product. The quality is poor, the maerial is dated, or it is not relevant to NASA's Science Mission Directorate.