Can you Trust ChatGPT for your Assignment – Testing & Reporting

Example:1 ChatGPT Provides Information About Systematic Review

Input by ResearchProspect: What is a systematic review?

ChatGPT Responds:

A systematic review is a type of research study that aims to synthesise all available evidence on a particular research question in a structured and systematic way. It involves a rigorous and transparent process of searching for, selecting, appraising, and summarising relevant studies.

The goal of a systematic review is to provide an unbiased and comprehensive summary of the available evidence on a particular topic, with the aim of answering a specific research question or informing clinical practice.

Systematic reviews can be conducted in any field of research, and can include various types of studies, such as randomised controlled trials, observational studies, and qualitative research. They are often used in healthcare to inform clinical practice guidelines and policy decisions, but can also be used in other fields such as education, social sciences, and environmental sciences.

Systematic reviews are considered the highest level of evidence in the hierarchy of evidence-based practice because of the rigorous and transparent methods used to synthesise the available evidence.

Example:2
Input by ResearchProspect: Can you provide a quick example of a systematic review?
ChatGPT Responds:
Sure, here’s an example of a systematic review:
Title: The effectiveness of cognitive-behavioral therapy for anxiety disorders: A systematic review and meta-analysis
Objective: To evaluate the effectiveness of cognitive-behavioral therapy (CBT) in the treatment of anxiety disorders.
Methodology: The authors conducted a systematic review and meta-analysis of randomised controlled trials that compared CBT to other treatments or to no treatment for adults with anxiety disorders. They searched several databases for relevant studies, assessed the quality of the included studies, and performed a meta-analysis to estimate the overall effect size of CBT.
Results: The authors identified 42 studies that met their inclusion criteria, including a total of 2,644 participants. The meta-analysis revealed that CBT was significantly more effective than other treatments or no treatment in reducing symptoms of anxiety, with a moderate effect size (Cohen’s d = 0.64). The effect size was larger for studies that used individual rather than group therapy, and for studies that focused on specific anxiety disorders rather than multiple disorders.
Conclusion: The authors concluded that CBT is an effective treatment for anxiety disorders, with moderate to large effect sizes observed across a range of anxiety disorders and treatment modalities. They recommended that CBT be considered as a first-line treatment for anxiety disorders.
The inherited AI technology allows ChatGPT to provide answers according to the users’ needs. For example, we first asked ChatGPT for a general response, followed by a more specific request for a simpler explanation.

Example:1 Chat GP Explains the Theory of Gravity.
Input by ResearchProspect: Can you quickly explain the theory of gravity?
ChatGPT Responds:
Sure, the theory of gravity is a scientific explanation of the force that attracts objects with mass towards each other. According to this theory, any two objects in the universe exert a gravitational force on each other, with the strength of the force depending on the masses of the objects and the distance between them.
The theory of gravity was first formulated by Sir Isaac Newton in the 17th century, and was later refined by Albert Einstein’s theory of general relativity in the 20th century. According to the theory of general relativity, gravity is not just a force between two objects, but a curvature of spacetime caused by the presence of mass and energy.
The theory of gravity has been extensively tested and has been shown to accurately predict the behavior of objects in our universe. It is the basis for our understanding of the motion of planets and stars, as well as the behavior of objects on Earth.

Example:2
Input by ResearchProspect: This is too complex for me. Can you explain gravity in simpler terms?
ChatGPT Responds:
Sure! Gravity is the force that pulls things towards each other. Everything in the universe that has mass or energy is affected by gravity. The bigger an object is, the more gravity it has. For example, the Earth has a lot of gravity, which is why we stay on the ground instead of floating off into space.
Gravity is what keeps the planets in our solar system orbiting around the sun. The sun’s gravity is so strong that it pulls the planets towards it, but the planets are moving so fast that they don’t fall into the sun.
Gravity is also what makes things fall to the ground when we drop them. When we drop something, it falls because the Earth’s gravity is pulling it towards the ground.
In summary, gravity is the force that pulls things towards each other, and it affects everything in the universe that has mass or energy.

Example: ChatGPT Limitations
ResearchProspect asks: How many countries start with the letter V?
ChatGPT answers: There are no countries in the world that start with the letter “V”.
As we can see, the ChatGPT tool suggests that no countries start with the letter “V”, even though ChatGPT still maintains its confident tone as if the information it provided was correct, which it clearly wasn’t.
Because we know that some countries start with the letter “V”, including Venezuela and Vietnam, it is vitally important to validate the answers ChatGPT offers to ensure they are credible. We recommend that students compare the answers against authentic sources to be confident of their authenticity.

Example:1 Chat GPT Learning From its Mistakes
ResearchProspect: But there are countries that start with the letter V.
ChatGPT answers: I apologise for my earlier response. You are correct, there are countries that start with the letter “V”. There are two countries in the world that start with the letter “V”:
1. Vanuatu – a small island nation located in the South Pacific Ocean.
2. Vatican City – a small city-state located within Rome, Italy, and is the headquarters of the Roman Catholic Church.
I hope this information is helpful!
We concluded that ChatGPT could not be relied upon because it can provide incorrect answers. However, we also found that ChatGPT can understand user feedback and incorporate it into the answers because it showed it could update its answers based on the feedback.
However, another education company tested ChatGPT’s understanding and learning capabilities through the following questions. It concluded that ChatGPT could also make mistakes in understanding feedback and continue giving the same output even if it recognised the mistakes.

Example:2 ChatGPT answers incorrectly
Company: Are there any singular nouns ending in a single “s”?
ChatGPT: Yes, there are singular nouns that end in just one “s”. Here are some examples:
● Glass
● Grass
● Bass
● Gas
● Pass
● Class
Note that these nouns can still be made plural by adding “es” to the end, such as “glasses”, “grasses”, “basses”, etc.

Example:3 ChatGPT responds to a correction
Jack: Most of these examples end in a double “s.” ChatGPT: I apologise for the mistake. You are correct, most of the examples I gave end in a double “s.” Here are some examples of singular nouns that end in a single “s”:
● Cross
● Boss
● Loss
● Kiss
● Bliss
● Dress
● Guess
● Address
These are just a few examples. There are many more singular nouns that end in a single “s.”

ResearchProspect asked: “Is London the Capital City of Birmingham?”
ChatGPT responds: No, London is not the capital city of Birmingham. London is the capital city of England and the United Kingdom. Birmingham is a city in central England, but it is not the capital city of England or the United Kingdom. The capital of England and the UK is London, and it is where the UK’s government and monarchy are located.
The AI model doesn’t have the information; rather, it develops responses from the patterns it worked with. It tells you what it knows but does not know what is true or false. When the user asks an unusual question that it didn’t encounter previously, the answers will sometimes sound made up and self-contradicting.
ChatGPT is likely to provide correct answers to most of the questions, but when the users ask strange questions they do not know about, it starts to fabricate the answers. Moreover, our research found that ChatGPT was only trained on data up to 2021, which means some of the responses will be outdated or unreliable.