| Campus | GroupLabel | TestID | Module | TestType | SimulatedActualResult | PassFail | Priority | QuickInsight |
|---|---|---|---|---|---|---|---|---|
| Melbourne | Mel g6 | SRCH_P001 | Product Search | Positive | "Product as PASS" | PASS | High | Standard positive test. |
| Melbourne | Mel g6 | AUTH_N0X | Authentication | Negative | Verify syst "Pass" | FAIL | Medium | Password length restriction is not enforced during login |
| Melbourne | Mel g6 | AUTH_N0X | Authentication | Negative | Verify logi shou | FAIL | High | Login restriction not triggered after repeated failed attempts |
| Melbourne | Mel g6 | SRCH_B001 | Product Search | Boundary | Verify syst system cra | FAIL | High | No input validation for search string length; system crashes |
| Melbourne | Mel g6 | DB_E001 | Database | Exception | Ensure sys User seei | FAIL | High | System fails to gracefully handle DB outage; user sees crash/error dump |
| Online | Onl g6 | GH-UT-SRH-P01 | Product Search | Positive | Verify pro search res | PASS | Medium | Standard positive test. |
| Online | Onl g6 | GH-UT-AU-N01 | Authentication | Negative | Verify syst Login is de | FAIL | High | Password policy not enforced; system accepts passwords under 8 characters. |
| Online | Onl g6 | GH-UT-SRH-N01 | Product Search | Negative | Verify the a validatio | FAIL | High | Missing input length validation triggers application crash on long queries. |
| Online | Onl g6 | GH-UT-AU-B01 | Authentication | Boundary | Verify logi Login is su | PASS | Medium | Standard boundary test passed. |
| Online | Onl g6 | GH-UT-DB-E01 | Database | Exception | Verify the a user-frie | FAIL | High | Database outage not gracefully handled; unhandled exception leads to blank/unresponsive UI. |
| Sydney | Syd g2 | (Pla AUTHENTI | Authentication | Positive | Test the us The user lc | PASS | High | Standard positive test. |
| Sydney | Syd g2 | (Pla AUTHENTI | Authentication | Negative | Check that The user ci | FAIL | High | Failed account status check (e.g., inactive). |
| Sydney | Syd g2 | (Pla AUTHENTI | Authentication | Negative | The user a After three | FAIL | High | Failed brute-force lockout. |
| Sydney | Syd g2 | (Pla SEARCH_P | Product Search | Boundary | Character System did | FAIL | Medium | Failed boundary length validation. |
| Sydney | Syd g2 | (Pla DB_E | Database | Exception | Lost datab Detect losi | FAIL | High | Failed graceful DB error handling. |
| Sydney | Syd g3 | (Mo SRCH_001 | Search | Positive | Verify pro Results ret | PASS | Medium | Standard positive test. |
| Sydney | Syd g3 | (Mo AUTH_002 | Authentication | Negative | Reject pas: 'Pas' | FAIL | High | Failed password policy (length/complexity). |
| Sydney | Syd g3 | (Mo AUTH_003 | Authentication | Negative | Lockout at Access der | FAIL | High | Failed brute-force lockout. |
| Sydney | Syd g3 | (Mo SRCH_004 | Search | Boundary | 255 accepi 255 OK; 25 | FAIL | Medium | Failed at 256-char boundary (crash). |
| Sydney | Syd g3 | (Mo DB_005 | Database | Exception | Handle DB Friendly m | FAIL | High | Failed graceful DB error handling. |
| Sydney | Syd g4 | AUTH_001 | Authentication | Positive | Session tim User loggE | FAIL | High | Missing session timeout; security risk on unattended kiosks |
| Sydney | Syd g4 | AUTH_002 | Authentication | Negative | Reject pas: 'Error: ' | FAIL | High | Password length validation missing; weakens authentication |
| Sydney | Syd g4 | SEARCH_P01 | Product Search | Negative | Handle quc Product qu | FAIL | High | SQL LIKE apostrophe not escaped; query breaks |
| Sydney | Syd g4 | SEARCH_B01 | Product Search | Boundary | Query lengr Show validi | FAIL | High | Boundary input not validated; potential denial-of-service |
| Sydney | Syd g4 | DB_E01 | Database | Exception | Handle Poi Friendly Err | FAIL | High | Missing exception handling and user messaging on DB timeout |
| Melbourne | Mel_gX | AUTH_001 | Authentication | Positive | The system logs the user in | PASS | High | Standard positive test. |
| Melbourne | Mel_gX | AUTH_002 | Authentication | Negative | System allows login despite the account being inactive | FAIL | High | Failed account status check. |
| Melbourne | Mel_gX | AUTH_003 | Authentication | Negative | System allows login even after 3 failed attempts | FAIL | High | Failed brute-force lockout. |
| Melbourne | Mel_gX | PROD_004 | Product Search | Boundary | The system searches successfully; empty list returned | PASS | Medium | Tested a passing boundary. |
| Melbourne | Mel_gX | DB_001 | Database Connectivity | Exception | System crashes or throws unhandled exception | FAIL | High | Failed graceful error handling. |
| Brisbane | Bri_gE | AUTH_SessionExpiry_001 | Authentication | Positive | Session expires and user is logged out | Pass | High | Testing the intended session timeout. |
| Brisbane | Bri_gE | AUTH_Login_001 | Authentication | Negative | Login succeeds on the fourth attempt | Fail | High | Failed brute-force lockout. |
| Brisbane | Bri_gE | SEARCH_Product_001 | Product Search | Negative | System crashes... "Unhandled input overflow" | Fail | High | Failed graceful handling of long input. |
| Brisbane | Bri_gE | SEARCH_QueryLimit_001 | Product Search | Boundary | System freezes... when 256 chars... entered | Fail | High | Failed at the boundary limit. |
| Brisbane | Bri_gE | DB_ConnectionLoss_001 | Database Connectivity | Exception | System freezes, no message is shown | Fail | High | Failed graceful error handling. |
| Rockhampton | Roc_gA | SEARCH_001 | Search | Positive | No results returned for apostrophe query | FAIL | High | Failed apostrophe handling. |
| Rockhampton | Roc_gA | AUTH_001 | Authentication | Negative | Login accepted with short password | FAIL | High | Failed password length policy. |
| Rockhampton | Roc_gA | AUTH_002 | Authentication | Negative | Fourth login succeeded; no lockout triggered | FAIL | High | Failed brute-force lockout. |
| Rockhampton | Roc_gA | SEARCH_003 | Search | Boundary | Query executed correctly | PASS | Medium | Passed 255-char boundary. |
| Rockhampton | Roc_gA | DB_001 | Database | Exception | System froze and failed to recover | FAIL | High | Failed graceful error handling. |
| Campus | GroupLabel | CaseReviewed | Improved_Test_1 | Improved_Test_2 | Improved_Test_3 | Rejected_Suggestion_1 | Rejection_Justification_1 | Rejected_Suggestion_2 | Rejection_Justification_2 | AddedTest_ID | AddedTest_Insight |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Melbourne | Mel g6 | AUTH_N0X (Lockout) / DB_E001 | Add CAPTCHA after 3 failed logins | Improve error message clarity (DB outage) | Introduce backend logging for DB errors | Lock duration test deferred to integration testing | AI suggested adding test for specific lock duration (e.g. 15m), but this is integration/config test, not unit test. | Background service check | AI suggested checking non-essential bg services; rejected as out of scope for this unit test. | AUTH_B0X | Boundary test for max password length |
| Online | Onl g6 | GH-UT-SRH-N01 (Ranking) / GH-UT-DB-E01 (Ranking) | Add explicit check for HTTP status 500 | Add max char test (256) for search | Specify error msg content in Expected Result | Deferred Accessibility assertions | AI suggested adding accessibility checks (contrast, font size) to DB error msg. Rejected as accessibility testing is scheduled for separate review cycle. | System-wide resilience test | AI suggested a full system recovery test, which is out of scope for this unit-level task and belongs in integration testing. | GH-UT-DB-N01 | Negative test for SQLi in search bar |
| Sydney | Syd g2 | AUTHENTI (Lockout) / DB_E (DB fail) | Install lockout mechanism | Lock frontend (not just backend) | Add CAPTCHA | No lock after 3 attempts | AI suggested this might be OK for usability; rejected as security risk. | Silent auto-truncation (transparency) | AI suggested silently truncating long search (SEARCH_B); rejected as it hides potential errors from user. | AUTHENTI | Negative test for session timeout |
| Sydney | Syd g3 | SRCH_001 (SLA enforce) / DB_005 (Concurrency) | Add SLA response time (e.g. <2s) | Add log/alert for DB connection pool exhaustion | Verify graceful recovery from deadlock | Chaos testing (deferred to system level) | AI suggested full chaos testing; rejected as this is a unit test, not system/resilience test. | Keep SRCH_001 priority Medium (business-driven) | AI suggested High priority for SLA test; rejected as occasional slow search is Med busines impact vs. security (High). | DB_006 | Recovery / Exception test for DB rollback |
| Sydney | Syd g4 | AUTH_002 (MFA scope) / DB_E01 (Recovery) | Add clear error msg for length fail | Enforce length validation (backend) | Simulate packet loss (DB recovery) | MFA out of project scope | AI suggested adding MFA test; rejected as MFA not in original project scope for A3. | Full disaster simulation out of unit scope | AI suggested full DR test; rejected as this is unit test, not system-wide DR test. | DB_B01 | Boundary test for DB connection pool limit (e.g. 101 connections) |
| Melbourne | Mel_gX | AUTH_002 | Enhance Auth Negative Tests (lockout) | Add Partial Database Failure Handling | Introduce Input Boundary Checks | Lowering priority of DB_001 to Medium | Database connectivity directly affects... critical kiosk functionality, so High priority remains essential. | Removing simulated actual results for AUTH_002 | Including simulated results highlights gaps... crucial for guiding bug fixes | AUTH_004 | Added "Session Expiration Test" based on AI feedback. |
| Melbourne | Mel_gX | DB_001 | Enhance Auth Negative Tests (lockout) | Add Partial Database Failure Handling | Introduce Input Boundary Checks | Lowering priority of DB_001 to Medium | Database connectivity directly affects... critical kiosk functionality, so High priority remains essential. | Removing simulated actual results for AUTH_002 | Including simulated results highlights gaps... crucial for guiding bug fixes | N/A | N/A |
| Brisbane | Bri_gE | AUTH_SessionExpiry_001 | Clarify Session Timeout Tolerance | Use Realistic Retail Credentials | Add Edge Case for Activity Reset | Include CAPTCHA in Lockout Test | Rejected: CAPTCHA would degrade usability... Kiosk design uses touch-based interfaces... aligns better with retail workflows. | Test Multiple Tabs for Session Expiry | Rejected: Kiosk systems operate in a controlled single-session environment... This scenario is irrelevant for our architecture. | AUTH_SessionReset_001 | Added "Session timer resets after user activity" based on AI edge case feedback. |
| Brisbane | Bri_gE | AUTH_Login_001 | Clarify Session Timeout Tolerance | Use Realistic Retail Credentials | Add Edge Case for Activity Reset | Include CAPTCHA in Lockout Test | Rejected: CAPTCHA would degrade usability... Kiosk design uses touch-based interfaces... aligns better with retail workflows. | Test Multiple Tabs for Session Expiry | Rejected: Kiosk systems operate in a controlled single-session environment... This scenario is irrelevant for our architecture. | N/A | N/A |
| Rockhampton | Roc_gA | AUTH_002 | Add Observability assertions (logs/metrics) | Use Deterministic fault injection profiles | Add Layered security validation (rate limits) | Full replacement of manual review | Rejected: regulatory validation requires human oversight... AI cannot judge legal text... ensures compliance with PCI-DSS. | Auto-fixing production config from failures | Rejected: Automatic updates could cause cascading system faults... preserves approval workflows... change management. | DB_002 | Added "Verify automatic recovery" after DB outage. |
| Rockhampton | Roc_gA | DB_001 | Add Observability assertions (logs/metrics) | Use Deterministic fault injection profiles | Add Layered security validation (rate limits) | Full replacement of manual review | Rejected: regulatory validation requires human oversight... AI cannot judge legal text... ensures compliance with PCI-DSS. | Auto-fixing production config from failures | Rejected: Automatic updates could cause cascading system faults... preserves approval workflows... change management. | N/A | N/A |
| Campus | GroupLabel | Claimed_ManualEffortReduction_% | ValuableCapabilities_Top2 | StatedLimitations_Top2 | HumanEssentialScenario | QuickInsight |
|---|---|---|---|---|---|---|
| Melbourne | Mel g6 | 65% | Automated script generation; Boundary data generation | Weak business context understanding; Cannot assess physical kiosk interactions | Usability/accessibility evaluation for elderly users | Human-led accessibility/usability/context is essential. |
| Online | Onl g6 | 70% | Automated JS unit tests; Boundary data generation | Misses business intent; Weak API integration testing | Empathy/accessibility and UX judgement at kiosk | Human-led accessibility/usability/context is essential. |
| Sydney | Syd g2 | 60-70% | Automated JS password tests; Boundary data generation (Search) | Instability across external API integrations | UX/accessibility judgement at kiosk | Human-led accessibility/usability/context is essential. |
| Sydney | Syd g3 | 65-75% | Coverage analysis; Failure prediction from complexity | Instability across external API integrations | UX/accessibility judgement at kiosk | Human-led accessibility/usability/context is essential. |
| Melbourne | Mel_gX | 40-60% | Automated JS scripts; Boundary data generation | AI may not fully understand business context or user experience nuances; AI cannot evaluate visual, tactile, or usability aspects | Human testing remains essential for exploratory usability and accessibility assessments. | Human-led accessibility/usability/context is essential. |
| Brisbane | Bri_gE | 60-85% | Test script generation; Test data creation; Coverage checking | don’t fully understand business rules; AI tests can fail... if the app design... changes | Human testing is very important for checking usability and customer experience... emotions, accessibility, and ease of use. | Human-led accessibility/usability/context is essential. |
| Rockhampton | Roc_gA | 60-85% | Auth test generation; Boundary-value inputs; DB fault-injection | AI struggles with complex, multi-layered system states.; AI... may also mishandle non-deterministic data. | Human testers are essential for evaluating user experience, compliance, and ethical handling... AI cannot interpret regulatory meaning or tone... empathy... make human testing irreplaceable. | Human-led accessibility/usability/context is essential. |
| Campus | GroupLabel | FocusArea | Year | VenueType | Technique_Quant | AuthorNoted_Limitation | StudentCritical_SynthInsight |
|---|---|---|---|---|---|---|---|
| Melbourne | Mel g6 | Prioritisation | 2025 | Journal | Supervised ML (Risk ID acc. 94%) | Performance depends on quality/volume of historical data | Critique: Model's 94% accuracy is misleading; it fully depends on a mature, stable historical dataset, which a new project like ours lacks. |
| Online | Onl g6 | Self-Healing | 2024 | Journal | Dynamic locator substitution (Reliability 70%->90%) | Does not ensure semantic/business correctness | Critique: Self-healing locators (90% reliability) only fix the *symptom* (broken test), not the *cause*. It doesn't guarantee the business logic is still correct. |
| Sydney | Syd g2 | Prioritisation | 2024 | Journal | Risk-based ML model | Semantic/business correctness not guaranteed | Critique: Prioritisation based on code churn is useful but misses the *business risk* aspect. A simple change in an auth file is more critical. |
| Sydney | Syd g3 | Prioritisation | 2024 | Journal | Deep ensemble CNN (Accuracy 79.18%) | Semantic/business correctness not guaranteed | Critique: Deep ensemble CNNs are a 'black box.' This makes it impossible to justify *why* a test was de-prioritised, which fails regulatory/audit requirements. |
| Melbourne | Mel_gX | Automated Test Case Generation | 2024 | Journal | k-nearest neighbors (KNN) | dependence on high-quality historical testing data | A critical challenge not fully addressed by the paper is scalability with large and heterogeneous datasets. |
| Brisbane | Bri_gE | Automated test case generation | 2024 | Journal | Deep Learning with domain adaptation | bounded by dataset size, domain specificity | How assertion generation scales... and interacts with dynamic runtime states are not fully covered in the paper. |
| Rockhampton | Roc_gA | Automated Test Case Generation | 2024 | Journal | transformer-based AI; reinforcement learning | incomplete GUI support... computational demands | biggest challenge is stateful cross-component testing... doesn’t fully address how AI-generated tests should be reviewed and approved in regulated systems |
| Campus | GroupLabel | Misconception_Theme | HumanCriticalAspect_Theme | Quote_1 | Quote_2 |
|---|---|---|---|---|---|
| Melbourne | Mel g6 | AI can fully replace human testing | Usability/accessibility require human judgement | AI excels at syntax but misses application context. | Only humans can assess what feels broken from a user's perspective. |
| Online | Onl g6 | AI 'thinks' like a human | Context, ethics, and UX sense-making | AI follows patterns, output not intentions. | Humans connect technical output with business meaning. |
| Sydney | Syd g2 | AI replaces human testers | Context, usability, and ethics require human judgement | AI follows patterns; humans connect to business meaning. | Usability and accessibility remain human-led. |
| Sydney | Syd g3 | AI replaces human testers | Context, usability, and ethics require human judgement | AI follows patterns; humans connect to business meaning. | Usability and accessibility remain human-led. |
| Melbourne | Mel_gX | AI can fully replace human judgment | evaluating contextual usability and business-critical logic | AI... relies on human expertise to... ensure business-relevant coverage. | Human testers are uniquely capable of understanding context... judgment and empathy, elements that AI cannot replicate. |
| Brisbane | Bri_gE | AI can replace human judgment | Context, usability, and business rules | AI can’t understand every special case, like kiosk timeouts or customer rewards. | Humans can notice... emotions, accessibility, and ease of use. |
| Rockhampton | Roc_gA | AI replaces human testers | Compliance, ethics, and UX | AI struggles with complex, multi-layered system states. | Human reasoning, empathy, and policy understanding... AI cannot interpret regulatory meaning or tone. |