How can I hook into a tokenizer to parse additonal content when it gets to a step before it checks for a newline.

[kelvinhammond]

Hello,

I'm trying to parse some markdown files which contains content like so:

```javascript
console.log("Hello world")
```{{execute}}

The codeFenced fails to parse the ` because it's not a newline at the end. What is the proper way to hook into the effects.attempt for this? Or should I go another direction. I was hoping to avoid re-implementing codeFenced if possible. I need to do the same for codeText (ex: `echo hello`{{execute}})

[wooorm]

You can’t hook into it like that.

Did you see this chapter? You probably shouldn’t create extensions.

[kelvinhammond]

I read that chapter, I've been over the README.md a lot while working on this. It's something I need to parse because I've a lot of older saved markdown files with this format which used an old unmaintained markdown parser. I see no other way to parse these files besides an extension.

[kelvinhammond]

I've written a regular expression (ish) parser in the past. I understand how the parsing works. I'm wondering if hacking the codeFenced and codeText to process characters after is the right way or if I should properly re-implement these parsers in my own extension and disable the codeFenced. I'll then create fromMarkdown and toMarkdown extensions for use with remarkjs.

[wooorm]

The solution to “I have old markdown that depends on an unmaintained markdown parser”, is to on your own start maintaining that markdown parser. I mean, do your thing, but as someone who maintains markdown parsers: that’s a ton of work over the years for one person.

I recommend migrating to more standard-ish markdown.
E.g., in CM you can put anything you want in the code fence. For example:

```js eval
console.log(1)
```

[kelvinhammond]

It's not really possible to migrate them all. What if I create a regex to preprocess the text with a regex and move the execute in the metadata?

[wooorm]

I would do the migration with some regexes too probably yeah!
Probably in a little Node script to do it if its lots of files and so you have loops, instead of regexes in your editor.