Cannot invoke "org.htmlunit.corejs.javascript.Scriptable.getParentScope()" because "obj" is null

[KiithNabaal]

Hi everyone!

I am super excited to see there is something compatible with Selenium that I can use to unit test with. My test works as follows:

In the constructor, I initialize a new WebDriver:

HtmlUnitDriverOptions options = new HtmlUnitDriverOptions(BrowserVersion.CHROME);

options.setJavaScriptEnabled(true)
       .setCapability(HtmlUnitOption.optThrowExceptionOnScriptError, false);

webDriver = new HtmlUnitDriver(options);

Then later on in a different method call, I instruct it to load a static resource. In this case, it is a rendered HTML file from twitter. I then immediately after instruct it to wait for the JavaScript to finish executing by waiting for the root node to finish rendering.

String currentDirectory = System.getProperty("user.dir");
webDriver.get("file:///" + currentDirectory + "/resources/test/web/twitter_source.html");

WebDriverWait jsWait = new WebDriverWait(webDriver, Duration.ofSeconds(20));
jsWait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.id("react-root")));

This all works well.

Then even later in the code, I run a JavaScript file that crawls the DOM and adds style attributes to each node. For example:

style-boundingx

When HtmlUnit processes this file, I get the exception or error message in the issue title:

Cannot invoke "org.htmlunit.corejs.javascript.Scriptable.getParentScope()" because "obj" is null

I tried to look up the javadoc for ScriptableObject but I get a 404 and so I cannot debug further. Any indication as to what the issue may be? I don't know what "obj" refers to. This code runs in Java Selenium with a ChromeDriver v127, so the JavaScript script should be fine. This only happens on HtmlUnit.

https://www.htmlunit.org/apidocs/org/htmlunit/corejs.javascript.ScriptableObject.html

Any help is appreciated. Thanks!

[rbri]

Can you please post the whole stacktrace.

[KiithNabaal]

2024-09-13 09:26:58,225 ERROR [main] fugu.web.CrawlerHelper: Failure crawling https://www.twitter.com
java.lang.NullPointerException: Cannot invoke "org.htmlunit.corejs.javascript.Scriptable.getParentScope()" because "obj" is null
at org.htmlunit.corejs.javascript.ScriptableObject.getTopLevelScope(ScriptableObject.java:1971)
at org.htmlunit.corejs.javascript.ScriptableObject.getTopScopeValue(ScriptableObject.java:2477)
at org.htmlunit.corejs.javascript.ClassCache.get(ClassCache.java:73)
at org.htmlunit.corejs.javascript.JavaMembers.lookupClass(JavaMembers.java:785)
at org.htmlunit.corejs.javascript.NativeJavaObject.initMembers(NativeJavaObject.java:64)
at org.htmlunit.corejs.javascript.NativeJavaObject.(NativeJavaObject.java:54)
at org.htmlunit.corejs.javascript.NativeJavaObject.(NativeJavaObject.java:45)
at org.openqa.selenium.htmlunit.AsyncScriptExecutor.createInjectedScriptFunction(AsyncScriptExecutor.java:147)
at org.openqa.selenium.htmlunit.AsyncScriptExecutor.execute(AsyncScriptExecutor.java:75)
at org.openqa.selenium.htmlunit.HtmlUnitDriver.executeAsyncScript(HtmlUnitDriver.java:846)
at fugu.web.browser.MockBrowser.executeAsyncJavaScript(MockBrowser.java:50)
at fugu.web.CrawlerHelper.getStyledDom(CrawlerHelper.java:463)

That last line is where my code begins

[KiithNabaal]

While looking through the output, I also saw this which might be relevant:

org.htmlunit.ScriptException: missing ; before statement (injected script#1)
at org.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:947)
at org.htmlunit.corejs.javascript.Context.call(Context.java:586)
at org.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:481)
at org.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:315)
at org.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:731)
at org.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:770)
at org.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:955)
at org.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:919)
at org.openqa.selenium.htmlunit.AsyncScriptExecutor.createInjectedScriptFunction(AsyncScriptExecutor.java:143)
at org.openqa.selenium.htmlunit.AsyncScriptExecutor.execute(AsyncScriptExecutor.java:75)
at org.openqa.selenium.htmlunit.HtmlUnitDriver.executeAsyncScript(HtmlUnitDriver.java:846)
at fugu.web.browser.MockBrowser.executeAsyncJavaScript(MockBrowser.java:50)
at fugu.web.CrawlerHelper.getStyledDom(CrawlerHelper.java:463)

I did a quick look through the JavaScript file a few times and didn't notice any semicolon missing, but I also wasn't being meticulous about it.

[rbri]

Is it possible to share the script or send it by private mail to me?
Currently I am on vacation but i can have a look when I'm back.

[KiithNabaal]

Talking with the team now, but this isn't an urgent matter or anything.

[KiithNabaal]

Sent it to the email you listed on your profile. Take your time since it is no rush. Enjoy your vacation 👍

[rbri]

Got it, this is really intresting because what you like to achieve is possible in HtmlUnit without injecting js at all (at least from a first look).

This will be a great thing to fix...