Brett Dean, perhaps Australia's pre-eminent composer and certainly one of its most productive, is personable, witty, and engaging. He talks with heartfelt eloquence about his work, but always with a refreshing directness and clarity that illuminates rather than obscures. Humour, too. This takes a special talent when the subject is Hamlet. The gloomy Dane is uppermost in Dean's mind at present, as he is fully occupied orchestrating the full score of his operatic version of the play, which will receive its première at the Glyndebourne Festival in 2017. Colouring-in the score is, as Dean says, producing a simultaneous sense of wonder and achievement.

Cutting Shakespeare's longest work down to size has been quite a challenge for Dean and his librettist, the Canadian writer Matthew Jocelyn. 'To set or not to set, that is the question,' Dean says, chuckling down the phone. (And, yes, the original line is in the opera. 'It would be notable by its absence. It's going to be there in one form or another.')

Dean is speaking from his home in Berlin. It's Saturday morning, and Dean's wife, the artist Heather Betts, has already headed off to the local market to do the weekend shopping. Dean is joining her shortly. It's a ritual. 'Everyone is at the marketplace catching up,' says Dean. 'It's very primal, and has gone on since Shakespeare's time.'

Back to the Prince of Denmark. More than a few composers have attempted to adapt Hamlet – most famously Ambroise Thomas and, in the mid-twentieth century, Humphrey Searle – with varying success. What attracted Dean?

'I was reticent when it was first suggested in conversations in Denmark, appropriately enough with a Danish tenor we know who's a soloist at the Danish Opera. He was keen to be the prince himself at some stage in his career. I was fascinated by the idea, but it took a while to warm to it.' In the way was what Dean describes as 'This bowing-down-to-the-giant aspect', and having to remove 'that burden of awe from the equation'.

Things became more positive when Heather Betts, inspired by the idea, started a cycle of Hamlet paintings. 'She said to me, "What would Will himself have thought of it?" and he would have said "Go for it!"'.

That, Dean says, was the trigger. He began working in earnest with Matthew Jocelyn to create their own Hamlet. 'Already, there are three published editions, two of which were published during Shakespeare's own lifetime. By definition, any Hamlet on stage or screen is going to be some sort of conflation. Decisions have to be made on which direction or how much of it to use.'

The first thing composer and librettist did was sit down at Jocelyn's Toronto home and read aloud between them the whole text. This took five hours. 'That meant we would have to reduce it by three-quarters or four-fifths to get a suitable amount of text one could sing within a two-hour period.'

Brett Dean (photograph by Pia Johnson)Before the opera proper, Dean and Jocelyn worked on a smaller piece, for soprano and string quartet, called And once I played Ophelia. 'We took the opportunity to establish something of our working relationship and the style of the libretto itself.' By focusing on Ophelia, this twenty-minute dramatic scena incorporates not only her words, but words said to her by other characters (for example, Hamlet's 'Get thee to a nunnery'). 'This enabled us to get to grips with various characters – Hamlet himself in his love poem, but also Polonius and Gertrude.' And once I played Ophelia was performed in 2015 in Melbourne with the Australian String Quartet and soloist Allison Bell. It has also been recorded for Chandos, with Bell and the Doric String Quartet.

Dean and Jocelyn also got to grips with matters metrical. 'One of the questions I'm asked a lot is how do you cope with the iambic pentameters. The rhythms of it,' says Dean. 'Shakespeare is forever displacing the accents anyway: "to be ..." is an inverted iambic pentameter, anyway, with two inversions in the one line. And he's forever repositioning the accents so it almost becomes prose. Apart from anything else, there is the drama and beauty of the words. I'd like to say it just writes itself, but it not that easy. But it is a constant source of inspiration in its own right.'

With the music for the opera, the Ophelia-related material automatically found its way into the main piece. But central to the opera is what one could call, à la Tristan, the Hamlet Chord, which is heard through the piece and is at present Dean's constant companion. 'I started in a sense with this four-note chord that gives potential in various directions,' says Dean. 'I thought a nice way of symbolising Hamlet's sense of indecision was a chord that can go in on itself or can expand. There's a sort of diatonic implication that leads to the brightness and power of the major, but it can crumble inwards into a dissonant world as well.'

Glyndebourne itself, while not exactly Elsinore, had its part to play in the gestation of Hamlet. 'The aspect of performances at Glyndebourne is that they are centred around the long dinner interval,' says Dean. A few years ago, he discussed the idea of his opera with an old Glyndebourne hand, his friend Simon Rattle, chief conductor of the Berlin Philharmonic Orchestra. 'He said there were two things to remember. First, because of the long interval, you want the second half to be shorter than the first. Second, that Glyndebourne has one of the greatest choruses in opera, with incredible young voices that are up for anything. I've tried to explore that in the scoring, and I'm looking at the option of having some voices in the pit with the orchestra. It's not exactly a largely "people" story, but through Matthew's clever and canny libretto, we've found some spots to use the chorus to populate it.'

In almost every respect, Hamlet could not be more different from Dean's previous opera, Bliss, based on Peter Carey's novel, which was staged by Opera Australia in 2010 and since recorded. 'Well, they do share one common line, which Amanda [Holden, Bliss's librettist] put into Harry Joy's mouth when he's confronted with the full extent of the horror of his own family and damns them for all time. Amanda puts in "Angels and ministers of grace, defend us", which is what Hamlet exclaims when seeing his father's ghost.'

Dean says there are other similarities between Bliss and Hamlet. 'Certainly both works dealing with love, madness, death. Maybe that's drawing a long bow with similarities. But they're both dipping into the same pond of the human condition.'

Brett Dean is one of those rare musical creatures who, like Benjamin Britten before him and his contemporaries Thomas Adès and George Benjamin, is equally adept at composing, conducting, and playing. Which of these skills came first?

'Obviously, it started with playing,' he says. 'Even nowadays, I see playing as a central part of the musician I am, and there's something about the physical act that I find very vital. I can't really imagine composing without playing. That would be rather lonely.'

Dean's entries into composing and conducting occurred during his early days as a violist in the Berlin Philharmonic in the mid-1980s, when he played with a group called the Scharoun Ensemble. 'This was essentially an octet based on the Schubert Octet, with limited repertoire that ventured into commissioning new works. This gave me the experience of how to nut out a complex score and make it work without a conductor. But you had to have the conductor's skill of taking it to pieces, look for what you're listening for. That led me to thinking I'd like to try my own hand at some of this. One night, after I got back after a Scharoun rehearsal, I complained to Heather about some new piece we were working on. She said, "You know, you keep coming home saying you'd do that bit differently ... well, why don't you do it yourself?"'

Brett Dean (photograph by Pia Johnson)

The composing began at around the same time, when Dean was regularly improvising with a fellow Australian musician Simon Hunt, who was travelling with a rock band and whose fame as Pauline Pantsdown & Co was yet to come. 'At Simon's insistence or suggestion, I remember him saying, "Next time we get together, you bring along some ideas and we'll record them as well." That was the prod I needed; that I'd secretly been waiting for. So I thought, well, I'll bloody well try.'

Dean, who has never formally studied composition, says he writes music in his own way, on his own terms. He has, he says, always been conscious of having a 'voice' as a composer, with his own particular style of music.

'I do think that's the single most important thing,' he says. 'All over the world thousands of people are writing music, which is amazing. But what is your voice? What constitutes your take on the aural world? What drives you to want to write music for others to hear? I do think there's something recognisable about my music. I'd like to think so, and think that some of the success I've had is because of it. But what it is is hard to nut out.'

It's also keeping that voice independent and honest. 'I think the pieces I am less convinced by are the ones where I was trying to be someone else or live up to someone else's expectations of what music in the twenty-first century should be. Or that I didn't listen to the qualities that I know I possess myself. It is important to put one's hand up and say there is a lyrical and expressive side to how I live and breathe and express myself in music. So don't try to deny who you are. At the same time, I'm fascinated about finding new ways to create and play sound. That's when it becomes more important then being yourself. "To thine own self be true."'

At the same time, Dean says he always questions if he is an Australian composer or something more worldly that that. He recalls a recent discussion in Canada on Australian music. 'Is that even important? Perhaps some of the strongest music from Australian composers isn't strong because it sounds Australian, whatever that means. Should that be a defining factor or is it good or bad? I think a big part of whatever my voice might be, or why it's important, comes from being a performer. Somehow, the performing gesture has always been very pivotal to me. Having been a player myself makes engaging with performers enormously important to me.'

Dean's expectations of what twenty-first-century music is are embedded in his own reflections of how the expectations of society itself have changed.

'I do think that the twenty-first century has brought undeniable differences, certainly politically and socially, in a different time. Geopolitics have changed hugely and very quickly. And the technological revolution has been a game-changer in so many avenues of life that it's hard to ignore. Not all of this has been necessarily positive. I don't want to bash the IT age. I also am the beneficiary of that technology. I use it to communicate and do my work as well. Of course, it's changed everything in its own way.'

Brett Dean's final concert at the Australian National Academy of Music (ANAM)

He cites accessibility as the prime reason for this change. 'The long-term impact will be to break down the "grand composer" image of the past. Modern composers are getting out and doing it because it's possible for anyone with the feel for it to get a laptop, readily available software, and start making their own music. That's incredibly democratic. The great-composer-of-the-past tradition is much more grassroots. But what comes out, remains to be seen. It's not that it will always result in great works of art.'

Brett Dean is also a teacher, although he has not as much time as he once did. He still passionately believes in musical education – just as he did when he was the director of the Australian National Academy of Music, in Melbourne (2008–10). 'ANAM remains incredibly dear to my heart,' he says. 'It's one of those very special places. The kind that doesn't exist all that well elsewhere.'

Mind you, it's not that long ago that the existence of ANAM itself was under threat. In 2008, the then federal arts minister, Peter Garrett, withdrew the institution's $2.5 million annual government funding. It is thanks to a tireless campaign led by Dean and supported by thousands of music lovers that ANAM was saved. Dean, it seems, has not forgiven Garrett. 'My God! Garrett's biography contained an unbelievable, infuriating couple of paragraphs about ANAM ... so self-aggrandising ... what a difference he made. Well, look where the place is now – in much better shape. He would have seen the place closed.'

A few weeks ago, a string quartet comprising ANAM students, Affinity Collective, was in Berlin as part of a study tour. Dean organised a concert which included a movement of his String Quintet, the composer on viola. 'It was a delight to be able to rehearse with them, and to introduce them to a very knowledgeable older audience of friends and neighbours.'

Dean's passion to teach is driven by a story told to him years ago by the great choreographer Jiří Kylián, who had been in the Northern Territory. 'He saw an indigenous dance group and asked one of the elders what it is that motivates him. The man told him "I learnt it from my father and I have to teach it to my son".'